Inter-Item Associations for the Brazilian Version of the Deese/Roediger-McDermott Paradigm

The emotional content of words can affect both true and false memory performance. One hypothesis suggests that the effects of emotion on memory stem from the semantic cohesion of these words. Emotional words are better remembered because they are more inter-related than neutral words (semantic cohesion hypothesis). Although support for this assumption has been found in tasks that measure true memory, less is known about how the structure of lexical knowledge affects emotional false memories. This is partially due to the scarcity of norms that capture the pre-existing knowledge structure of verbal materials commonly used to investigate emotional false memories, such as the Deese/Roediger-McDermott word lists. In this study, we present inter-item association norms for the 44 lists of the Brazilian version of the DRM paradigm. Free-association responses were collected from a sample of 1,042 undergraduates and were used to estimate the level of connectivity among the words present in the DRM lists. Connectivity measures were then used to test the semantic cohesion hypothesis. No signifi cant correlations were found between the emotional measures (valence and arousal) and the connectivity measures. The results do not give support to the semantic cohesion hypothesis and suggest that, for the Brazilian version of DRM lists, inter-item association and emotionality can be independently manipulated.

Pre-existing knowledge plays an important role in memory.Information organized through years of learning can infl uence the ability to encode and retrieve new information (Nelson & Zhang, 2000;Nelson, Zhang, & McKinney, 2001;Schacter, Norman, & Koutstaal, 1998).For example, when people memorize a sequence of related  Endereço para correspondência: Programa de Pós-Graduação em Psicologia, Pontifícia Universidade Católica do Rio Grande do Sul, Av.Ipiranga, 6681, prédio 11, sala 940, Porto Alegre, RS, Brasil 90619-900.E-mail: lilian@ pucrs.br.This research was supported by grant 509833/2010-1 from CNPq (Brazil's National Council for Scientifi c and Technological Development) to Lilian M. Stein.words (e.g., door, pane, curtain, house), they use their world knowledge to cue recall.They may remember having seen household items in the list and correctly recall the word house.However, that same world knowledge may induce them to error.Participants often claim to have seen window, a word related to the studied items but absent from the original list (Deese, 1959b;Gallo, 2010;Roediger & McDermott, 1995).
Lists of associatively related words have been extensively used to investigate the structure of pre-existing knowledge (Cramer, 1968;Nelson, McEvoy, & Dennis, 2000;Nelson, McEvoy, & Schreiber, 2004).They have also been used to assess how previous knowledge can affect memory in laboratory tasks (Gallo, 2006).In fact, association norms can predict both true and false memories.Retrieval of a studied word (true memory) is more likely if the word is cued by a strong pre-experimental associate (i.e., a new word strongly related to the studied word) than if the word is cued by a weak associate (Deese, 1959b;Nelson & McEvoy, 1979).Likewise, retrieval of a new word (false memory) is more likely if it is strongly associated to the studied words than if it is weakly associated to the studied words (Gallo, 2006;McEvoy, Nelson, & Komatsu, 1999).
A particular set of association norms, the Deese/ Roediger-McDermott (DRM) lists (Deese, 1959b;Stadler, Roediger, & McDermot, 1999), proved popular among memory researchers (Gallo, 2010).This is partly due to the lists' reliability in generating false memories in the laboratory (Stadler et al., 1999).DRM lists were created by presenting participants with theme words (critical words) and asking them to produce the fi rst semantically related word that came to mind.The most frequent responses were then grouped into lists.The words in a DRM list are thus all associated to the critical word.In the DRM paradigm, the associates of the critical word, but not the critical word itself, are presented during a study phase.The phenomenon of false memories is demonstrated at test, when participants wrongly recall or recognize the critical word.Pre-experimental knowledge plays a role in the DRM paradigm as the probability of producing a false memory is strongly related to the associative strength between the study item and the critical word (Roediger, Watson, McDermot, & Gallo, 2001).
The DRM paradigm has also been used to study the role of emotion on the production of false memories (e.g., Brainerd, Stein, Silveira, Rohenkohl, & Reyna, 2008;Budson et al., 2006).The emotional contents of words can be assessed by asking participants to rate them in terms of valence (how pleasant the word is) and arousal (how exciting the word is).Valence and arousal are dimensions commonly used to quantify emotionality (Kensinger, 2004;Russell, 1980).Valence and arousal ratings can be incorporated into DRM lists, giving rise to emotional DRM lists.Research with emotional DRM lists has shown that negative lists elicit both more true and false memories than neutral lists (Brainerd, Holliday, Reyna, Yang, & Toglia, 2010;Brainerd et al., 2008;El Sharkawy, Groth, Vetter, Beraldi, & Fast, 2008).
One account suggests that negative words elicit more true and false memories because they are more inter-related than neutral words, as they share both categorical (e.g., "torture", "pain") and thematic membership (e.g., "gun", "corpse"; Maratos, Allan, & Rugg, 2000;McNeely, Dywan, & Segalowitz, 2004).Because unstudied negative words are more related to their studied counterparts than neutral words, they are more likely to be mistaken for a related studied word (in a recognition test) or generated by associates (in a recall test).True memories, on the other hand, stand to benefi t from semantic relatedness: Emotional words are better remembered than neutral words because they are more organized, in the same way that categorized words are better remembered than random words (Talmi & Moscovitch, 2004).
Support for this semantic cohesion hypothesis comes from experiments that manipulated conceptual similarity.In these studies, participants rated the level of association among words; semantic relatedness scores were then computed based on these ratings.When semantic relatedness was controlled, neutral words were recalled as well as negative words, suggesting that semantic similarity, not valence, drives the effect of emotion on true memories (Buchanan, Etzel, Adolphs, & Tranel, 2006;Talmi & Moscovitch, 2004).
It is less clear, however, whether semantic cohesion can also account for emotional false memories.Low levels of false memory in free-recall studies prevented a thorough assessment of this issue (Buchanan et al., 2006;Talmi & Moscovitch, 2004).In addition, confl icting results in recognition memory studies, which either supported (Maratos et al., 2000;Windmann & Kutas, 2001) or refuted the cohesion hypothesis (McNeely et al., 2004), indicate that this is still an open question.In particular, a recent study reported more false recall and false recognition for negative DRM lists than for neutral lists, despite a stringent control of semantic cohesion, suggesting that emotional valence can affect false memories over and above any effect of semantic cohesion (Dehon, Laroi, & Van der Linden, 2010).
The resolution of the debate concerning the specifi c roles of semantic cohesion and emotionality on memory performance may require the use of norms that capture the pre-existing knowledge structure of materials commonly used to investigate emotional false memories, such as emotional DRM lists.Norms are particularly important in this case because the assumption that negative words are more inter-related than neutral or positive words has not yet been established.In fact, a recent study has shown that, contrary to the common assumption, positive words may be more inter-related than negative words in semantic similarity tasks (Unkelbach, Fiedler, Bayer, Stegmuller, & Danner, 2008).If positive words are indeed more inter-related than negative words, then one would expect to fi nd more false memories for positive lures than for negative lures, a result not borne out by current data (Brainerd et al., 2008).To reconcile these seemingly contradictory results, it is thus important to determine the relative degrees of inter-item association for negative and positive DRM lists relative to neutral DRM lists.
Norms are also important for the semantic cohesion debate because they can help defi ne more precisely the nature of the relationship between pre-existing knowledge and memory performance.Different norms -free-association probabilities, similarity ratings, and co-occurrence statistics -are sensitive to different kinds of lexical relations (Maki & Buchanan, 2008;Nelson, Dyrdal, & Goodmon, 2005).Free-association probabilities, for example, are better predictors of cued recall than similarity ratings, probably due to the closer match between the association measure and the memory task (Nelson et al., 2005).Thus the type of norms used to select the stimuli (free association vs. ratings) and the type of memory task (recall vs. recognition) are relevant to test the semantic cohesion hypothesis.
Most studies that investigated the cohesion hypothesis, however, relied only on similarity ratings to measure semantic relatedness (e.g., Buchanan et al., 2006;McNeely et al., 2004;Talmi & Moscovitch, 2004).Moreover, the similarity ratings in those studies were obtained from small groups of participants in pilot tests (N < 20), not from standard norms, and were restricted to negative words, leaving undetermined the role of relatedness on memory for positive words.
The aim of this study is to present norms of inter-item association for the Brazilian version of the DRM paradigm (Stein, Feix, & Rohenkohl, 2006).The Brazilian DRM includes negative, positive and neutral lists (Santos, Silveira, Gomes, & Stein, 2009) and has been successfully used in memory studies (Brainerd et al., 2008;Grassi-Oliveira, Gomes, & Stein, 2011).We collected free-association responses to 660 words and used them to estimate both the number of intra-list associates for each DRM list (set size) and the strength of these inter-item associations (connectivity).We estimated free-association probabilities, rather than similarity ratings, because DRM lists have been constructed based on the former measure (Stadler et al., 1999;Stein et al., 2006).Inter-item association norms were then used to investigate the relationship between semantic cohesion and emotionality.
To provide converging evidence for the validity of the measures reported in this study, we also assessed whether inter-item connectivity correlates with recall performance.Previous research has shown that strongly connected lists are associated with more true recall and less false recall than weakly connected lists (Deese, 1959a;McEvoy et al., 1999, Experiment 2).If the measures presented here are good estimates of inter-item associations, then we would expect a similar pattern when comparing connectivity measures with recall performance from a previous study that has used the same stimuli normed here (Stein et al., 2006).

Participants
A total of 1,042 native Portuguese speakers (350 male, 692 female) participated in this study.Participants were college students from universities in the states of Rio Grande do Sul and Paraná (southern Brazil) aged between 16 and 52 years (M = 21.6,SD = 5.3).Most participants (80%) were aged between 18 and 26 years old.Only 2% were over 40 and 8% were under 18.Thus, the association norms reported here were based mainly on data from young adults.Participants within each university were sampled from 30 courses, including students from Humanities, Exact Sciences and Biological Sciences.Written informed consent was obtained from all participants and the study was approved by the university's Institutional Review Board (CEP: 05/02829, Pontifícia Universidade Católica do Rio Grande do Sul [PUC-RS]).

Materials
Stimuli consisted of 44 lists of 15 words drawn from the Brazilian version of the DRM paradigm (Stein et al., 2006).The words in each list were all semantically associated to a theme word.Normative measures of emotionality (Santos et al., 2009), forward strength, backward strength, set size, concreteness and word frequency (Stein & Gomes, 2009) for these stimuli have been previously reported.
Forward strength, backward strength and set size are measures of word association.Forward strength is the probability that a cue word (e.g., river) produces a target word (e.g., water).For DRM lists, the cue word is always the critical word and a list's forward strength is the average of the strengths between the critical word and their 15 associates.Backward strength is the probability that a target word produces its cue word.For DRM lists, backward strength refers to the frequency that an associate generates back its theme word; a list's backward strength is the average of the strengths between the 15 associates in a list and their critical word.Set size is the number of different associates of a cue word produced by two or more participants in a free-association task.
To collect measures of inter-item association, 15 sets of 44 unrelated words were created by randomly selecting words from the DRM lists.Each set contained only one word from each of the 44 lists.Thus the 15 sets covered all 660 words.The words in each set were printed in a sheet of paper and the sets were randomly distributed across participants.On average, 69 participants contributed freeassociation data for each word.

Procedure
Participants were tested in groups (approximately 25 participants per group).After signing a consent form, each participant received a sheet of paper with 44 entries.Each entry displayed a word (cue) followed by a blank space (e.g., water -____).Participants were instructed to write down in the blank space the fi rst word that came to mind after reading the cue.They were asked (a) to produce a word that was semantically related to the cue (e.g., book in response to library), (b) to write only one word per entry, (c) not to use the cue itself as a response, (d) to write legibly, and (e) to leave no entry unanswered.In order to stimulate the generation of responses, they were told not to worry about spelling and that there was no right or wrong answer.Participants returned the sheets as soon as they fi nished (self-paced task) and each data collection session took less than 30 minutes to be completed.

Data Analysis
Responses were pooled according to the following rules: (a) select a single form of nouns and adjectives varying in number [botão and botões were treated as botão (button)], gender [elástica and elástico were treated as elástica (elastic)] or diminutive-augmentative suffi xation [sopinha and sopão were treated as sopa (soup)]; (b) select the infi nitive form of verbs varying in subject agreement [pulo and pula were treated as pular (to jump)] and tense [escuto and escutou were treated as escutar (to listen)].
Associations between list items were quantifi ed using two measures defi ned in Nelson et al. (2004).The fi rst measure, called connectivity, refers to the number of associates that a word elicits within the same DRM list (only associates generated by at least two participants were included).For example, when the cue paper (a word belonging to the list pen) was given to participants, they generated several words, including notebook and to write, which also belong to the DRM list pen.In this example, the connectivity of paper was 2. By contrast, when the cue lid (also belonging to the list pen) was given to participants, none of the words they generated belonged to the list pen; the connectivity of lid was thus 0. Since each DRM list has 15 words, connectivity values ranged from 0 to 14.In this study, only the mean connectivity of a list is reported.Mean connectivity is the sum of connectivities of the 15 words in the list divided by 15.Connectivity values for individual words are provided as Supplemental Material (see Appendix).
The second measure of inter-item association, called connectivity strength, refers to the strength of association between pairs of words in the same DRM list.More specifically, connectivity strength is the probability that a word in a given DRM list elicits another word in the same list by free association.For instance, 3 out of 69 participants produced the word notebook when cued with the word paper; thus the connectivity strength between paper and notebook is 3/69 = .043.In this study, we report only the summed connectivity strength of a list.Summed connectivity strength is the sum of the connectivity strengths of the 15 words in a given list.For example, 5 out of 69 participants also generated to write in response to paper (connectivity strength = .072)and no other words belonging to the list pen were generated when participants were cued with paper.Thus, the summed connectivity strength of paper is .115(= .043+ .072).Strength measures for each word pair are provided as Supplemental Material.In addition to the inter-item association norms, the Supplemental Material also include measures of valence, arousal, concreteness, frequency, forward strength, backward strength and set size from the Brazilian DRM.We found this appropriate for two reasons.First, the analyses described below included norms from other word features, such as valence and arousal.Presenting these norms alongside the new inter-item association data should help clarify the results.Second, this is the fi rst study to present the Brazilian version of the DRM paradigm in English.By making these norms available to a wider audience, we hope to foster not only research with Portuguese speaking populations but also with cross-cultural samples.
Statistical tests were conducted using SPSS.A significance criterion (α) of .05 was adopted for all tests.

Results and Discussion
In the following, we fi rst describe the distribution of association measures (i.e., mean connectivity and summed connectivity strength) for the set of 44 lists of the Brazilian DRM lists.Next, we assess whether mean connectivity correlates with recall data from a previous study (Stein et al., 2006); the aim is to support the validity of the measures presented here by replicating a known fi nding.Finally, we test whether inter-item association correlates with positive and negative valence ratings, a fi nding that would be consistent with the semantic cohesion hypothesis.
Although mean connectivity and summed connectivity strength were highly correlated (Pearson's r = .78,p < .001), it is instructive to treat those measures separately as they convey different information about intra-list word associations.For example, the DRM list river has relatively low mean connectivity (1.067; more than half a standard deviation below the mean) but relatively high summed connectivity strength (2.628; more than half a standard deviation above the mean) in comparison to other DRM lists that have similar mean connectivity values, such as the list mountain (mean connectivity = 1.067; summed connectivity strength = 1.209).This discrepancy between mean connectivity and summed connectivity strength stem from strong associations among few list items.In the list river, two pairs of associates (waterfall-water = .424,fi shing-fi sh = .563)accounted for 37% of the list's summed connectivity strength.By contrast, in the list king it was necessary to sum the strength of six pairs of associates to account for the same share of the list's summed connectivity strength (nobility-wealth = .171,authority-power = .141,palace-wealth = .132,queencrown = .116,sovereign-power = .108,empire-power = .104;mean connectivity = 2.13; summed connectivity strength = 2.02).Thus, reporting mean connectivity and summed connectivity strength separately should facilitate stimulus selection by researchers interested in separating lists with many weak connections from lists with few strong connections.To support the validity of the association estimates reported in this study, we tested whether mean connectivity correlated with recall performance, regardless of emotionality.Previous studies have shown that true recall is higher and false recall is lower for highly connected lists (Deese, 1959a;McEvoy et al., 1999, Experiment 2).Studied words are more likely to be recalled in highly inter-connected lists because they can cue each other at test.Unstudied critical words are less likely to be falsely recalled in highly inter--connected lists because they compete with a larger set of cued associates, which decreases their chance of being elicited.We correlated mean connectivity estimates with the recall data reported by Stein et al. (2006), who used the same 44 DRM lists normed here.The results revealed a signifi cant positive correlation between mean connectivity and proportion of true recall (r = .46,p = .002)but no correlation between mean connectivity and proportion of false recall (r = -.04,p = .79).
These results partially replicated the fi ndings by Deese (1959a) and McEvoy et al. (1999) and provide some support for the validity of our connectivity estimates.The null correlation for false recall may be due to the narrower range of connectivity values in our lists compared to the more extreme values reported by McEvoy et al. (1999).In that study, 12 lists of associates were defi ned as highly connected if their connectivity values were greater than 2.50.By contrast, none of our lists reached that threshold; the greatest mean connectivity here was 2.27.To the extent that false recall depends on the competition between associates, the lack of extremely high connectivity values in our sample may have reduced competition and, consequently, the relationship between connectivity and false recall.
We now turn to the relationship between inter-item association and emotion.If semantic cohesion underlies the effects of emotional words on memory, then inter-item association should be related to valence in the following manner: (a) positive lists (valence ≥ 6) should be positively related to inter-item association measures; (b) negative lists (valence < 4) should be negatively related to inter-item association measures.There are 24 positive lists and 13 negative lists in the stimulus set.
Figure 1 shows the relationship between summed connectivity strength and valence (Fig. 1a) and between mean connectivity and valence (Fig. 1b).Visual inspection suggests that inter-item association measures and valence ratings are not related: Neither positive nor negative lists show a monotonic relationship with summed connectivity strength or mean connectivity.There is also no suggestion of a non-monotonic relationship.
(b) (a) Simple correlations confi rmed this pattern.For positive lists, there was no signifi cant correlation between valence and summed connectivity strength [r(22) = -.003,p = .98]and between valence and mean connectivity [r(22) = -.12,p = .58].For negative lists, there was also no signifi cant correlation between valence and summed connectivity strength [r(11) = .20,p = .50]and between valence and mean connectivity [r(11) = .13,p = .66].These results were obtained using the lists' mean valence ratings.Similar results were produced when the valence of the lists' critical words were used instead.
As with valence, there was no clear relationship between arousal and inter-item association: Neither summed connectivity strength [r(42) = -.12,p = .43]nor mean connectivity [r(42) = -.04,p = .80]correlated with arousal.When separate correlations were conducted with the arousal ratings of positive and negative lists, again no signifi cant relationship was found (all ps > .10).
To increase statistical power, we repeated the previous analyses including neutral lists.Neutral lists (valence between 4 and 5.99) were incorporated into the positive or negative groups, such that "neutral" lists with ratings between 5.01 and 5.99 were treated as "positive" lists and "neutral" lists with ratings between 4 and 4.99 were treated as "negative" lists.This classifi cation is reasonable in that participants judged the words' pleasantness ("positive", > 5) or unpleasantness ("negative", < 5) relative to the mid-point of the scale ("neutral", 5).For positive lists, there was no signifi cant correlation between valence and summed connectivity strength [r(26) = -.03,p = .88]and between valence and mean connectivity [r(26) = -.01,p = .95].For negative lists, there was also no signifi cant correlation between valence and summed connectivity strength [r(14) = .12,p = .65]and between valence and mean connectivity [r(14) = -.01,p = .98].There was also no clear relationship between arousal and inter-item association measures (all ps > .13).
Taken together, these results indicate that the number and strength of within-list connections are not related to the DRM lists' emotional features.Thus, no support for the semantic cohesion hypothesis was found for the 44 lists in this stimulus set.We are not claiming that the semantic cohesion hypothesis is incorrect.It may well be the case that the effect of emotion on true and false memories is mediated by the pre-experimental semantic relationships between the studied words.What these results indicate, however, is that inter-item connectivity and valence can be independently manipulated using this set of stimuli.
Two potential limitations of this study should be pointed out.The fi rst is that the norms presented here were obtained from a specifi c region in southern Brazil.Caution should thus be exercised when generalizing these norms to other Portuguese-speaking regions or countries, as word associations may be infl uenced by geographical and cultural specifi cities.However, the fact that robust false memory results were obtained with this stimulus set in other Brazilian regions (e.g., Huang & Janczura, 2008) suggests that the association properties captured in these norms are relatively stable across regions.The second potential limitation concerns the age range of the sample (16-52 years).Associative relations among words develop from early childhood to adulthood (Bjorklund, 1987), and such variability may explain differences in cognitive phenomena across age groups (e.g., Carneiro, Albuquerque, Fernandez, & Esteves, 2007).Because our sample included participants from different age groups (adolescents, young adults and middle-age adults), it could be argued that the norms presented here may refl ect properties from different populations.This is unlikely, however, as the bulk of our sample (88%) consisted of young adults (18-35 years).The small number of adolescents and middle-aged adults in the sample prevented us from obtaining reliable estimates of connectivity for these age groups.Further research may help clarify the issue of how word-association properties, such as intra-list connectivity, vary across age.

Conclusion
This study presented norms of intra-list connectivity and connectivity strength for the Brazilian version of the DRM paradigm, extending the semantic association norms previously obtained for this stimulus set (Stein & Gomes, 2009).Connectivity measures were then used to assess the role of pre-experimental knowledge on recall (Stein et al., 2006) and on the emotional properties of Brazilian DRM lists (Santos et al., 2009).The results indicated that inter-item association measures are independent of emotionality measures, suggesting that connectivity and emotionality can be factorially manipulated in future studies using this set of stimuli.The norms presented in this study may thus facilitate stimulus selection and stimulate research spanning the fi elds of semantic organization, false memories and emotion.

Figure 1 .
Figure 1.Scatterplots showing (a) the relationship between valence (mean valence of the words in each DRM list) and summed connectivity strength (sum of all pair strengths within each list); (b) the relationship between valence and mean connectivity (mean number of connections across the 15 items in each list).Each data point represents one of 44 DRM lists.

Table 1
Mean Connectivity (MC) and Summed Connectivity Strength (SCS) for the 44 Word Lists from the Brazilian Version of the Deese/Roediger-McDermott Paradigm (DRM lists)