Reliability of the Brazilian version of the Cambridge Cognitive Examination Revised CAMCOG-R

Confiabilidade da versão brasileira do Cambridge Cognitive Examination Revised - CAMCOG-R

ARTICLES

Reliability of the Brazilian version of the Cambridge Cognitive Examination Revised CAMCOG-R

Confiabilidade da versão brasileira do Cambridge Cognitive Examination Revised - CAMCOG-R

Emylucy Martins Paiva Paradela^I; Claudia de Souza Lopes^II; Roberto Alves Lourenço^I,III

^IGeriatrician, Service of Geriatrics, Public outpatient clinic Piquet Carneiro, Rio de Janeiro RJ, Brazil

^IIEpidemiologist, Professor of the Department of Epidemiology, Social Medicine Institute of Rio de Janeiro State University, Rio de Janeiro RJ, Brazil (UERJ)

^IIIGeriatrician, Professor of the Department of Internal Medicine, Faculty of Medical Sciences, UERJ

ABSTRACT

The study' goal was to evaluate the reliability of the Cambridge Cognitive Examination Revised-Brazilian version (Br-CAMCOG-R), a neuropsychological battery measuring the global cognitive function. It was applied on 123 elders and retested at a mean interval of 30.7 days; 60 were evaluated by two raters at the same time. The intraclass coefficient for the set of items and for the subscales varied from 0.93 to 0.98. In the retests the agreement was nearly perfect for the set of items. There was no expressive range in the stability of the instrument for sex, age, schooling, or for the presence of dementia. The Cronbach' alpha of the set of items of the test was high (0.89). The Br-CAMCOG-R has obtained a high level of stability with time, agreement among raters, and optimum internal consistency; it can be useful for epidemiological studies and in specialized clinics to evaluate cognitive functions in elders.

Key words: cognition, reliability, reproducibility of results, aged, dementia.

RESUMO

O objetivo deste estudo foi avaliar a confiabilidade da versão brasileira do Cambridge Cognitive Examination Revised (Br-CAMCOG-R), uma bateria neuropsicológica que mede a função cognitiva global. Ele foi aplicado em 123 idosos e reaplicado no intervalo médio de 30,7 dias; 60 destes idosos foram avaliados por dois aferidores ao mesmo tempo. O coeficiente de correlação intraclasse para o conjunto dos itens do teste assim como das subescalas variou de 0,93 a 0,98. No reteste a concordância foi quase perfeita para o conjunto dos itens. Não houve variação significativa na estabilidade do instrumento entre os sexos, idade, escolaridade ou presença de demência. O alfa de Cronbach para o conjunto dos itens do teste foi alta (0,89). O Br-CAMCOG-R teve altos níveis de estabilidade no tempo, entre os examinadores e ótima consistência interna; pode ser útil em estudos epidemiológicos assim como em clínicas especializadas na avaliação das funções cognitivas dos idosos.

Palavras-chave: cognição, confiabilidade, reprodutibilidade dos testes, idoso, demência.

The use of reliable and valid instruments to evaluate cognition of the elderly with memory complaints provides more accurate and reproductive diagnosis, making it possible for the health professional to be able to trace therapeutic goals based on objective results. It also allows comparisons between different populations, therefore helping to standardize procedures in clinical and epidemiological studies¹. Although there is no standard neuropsychological test for the diagnosis of dementia, exists several instruments used for this purpose. Neuropsychological batteries of tests are widely used to evaluate subjects who tested positive in the screening examinations. The lack of a specific biological marker for the diagnosis of dementia in the clinical practice amplifies the vitality of this evaluation². Among the brief neuropsychological batteries mentioned in the literature, the Cambridge Cognitive Examination (CAMCOG) has the advantage of offering a summary measurement of the global cognitive function, allowing changes at cutoff score according to age and schooling^3,4.

The Cambridge Cognitive Examination-Revised (CAMCOG-R) is the section B of the revised version of the Cambridge Examination for Mental Disorders of the Elderly (CAMDEX-R) which was developed at Cambridge University, in England⁵. It has incorporated a few screening instruments for cognitive disturbances widely used in epidemiological research, such as the Mini-Mental State Examination (MMSE)⁶, and 8 of the 10 items of the Hodgkinson Abbreviated Mental Test⁷. The CAMCOG-R is subdivided into 8 subscales, namely: orientation (time and place), language (comprehension and expression), memory (incidental, remote, recent, and new learning), attention, calculation, praxis, perception, and executive functions (ABSTRACT thinking, ideational fluency, and visual reasoning). The instrument is composed of 69 items with a maximum score of 105 points; the higher the total score, the better the performance in the test. The complexity of the instrument demands training for the raters and the copyright is reserved by the Cambridge University Press. Several reliability and validity studies analysing the first version of the instrument (CAMCOG) have been published^{3, 8-10}, the same being true of the revised version (CAMCOG-R)^11-15.In Brazil, the only study found by the authors which validates instruments from this country' perspective was the one written by Bottino et al.¹⁶, who made a translation and an adaptation into Portuguese of CAMDEX, and studied the reliability of this Brazilian version using 40 elderly subjects. The interrater reliability, which was evaluated using the ICC, proved to be high, varying from 0.79 to 0.99 for a pair of psychiatrists who applied the instrument, and from 0.67 to 1.0 for another pair (p<0.001).

The objective of the present study is to evaluate the interrater reliability and test-retest as well as the internal consistency of the Br-CAMCOG-R, as one of the vital steps in the evaluation process of the psychometric properties of this instrument.

METHOD

Design and population

This study took place in a public geriatric outpatient clinic in Rio de Janeiro, Brazil. The source population was composed of 1,200 elders who attended the clinic between May 2^nd and December 28^th, 2006. A research team examined the records of the subjects who would be treated and selected those who fulfilled the eligibility criteria. On the appointment day, the selected subjects were invited to participate in the study. In all, 196 subjects agreed to participate, gave their informed consent, and took the Br-CAMCOG-R. The first 60 subjects tested were also used to assess the interrater reliability of the instrument. On retest invitation 123 out of 196 subjects returned.

The inclusion criteria were: to be over 60 years of age, to be able to hear and to understand the survey' objectives, and to have obtained >14 points in the MMSE taken in the previous 11 months, since an averagely maintained cognition is required for wider neuropsychological testing. The exclusion criteria were: delirium, sensorial deficiency causing sight or hearing impairment, motor deficiency or tremor in the dominant hand which impaired tasks such as writing and copying, or any severe illness.

As for the evaluation of the interrater reliability, two raters applied the test and both of them wrote down the answers separately. The raters were not allowed to discuss their doubts, which were answered later by the coordinator of the survey. The test-retest reliability was evaluated with a mean interval of 30.7 days between the two ratings by the same rater.

Instrument application and the evaluated variables

The test application was done by one of the five trained examiners for the study: an occupational therapist, three psychologists, and a geriatrician. The testing took place in a reserved room, well lit, with little external noise, and with no visible calendars or clocks. Each subject submitted to the test sat on a chair at a table used as a surface upon which to write or draw, and in case he or she wore eyeglasses, was asked to wear them.

The patient' schooling was defined as completed years of school in the formal educational system as informed by the subject (failed school years were not taken into consideration). They were then divided in the following categories: no formal schooling/illiterate, low schooling (1 to 4 years of schooling), medium schooling (5 to 8 years of schooling), and high schooling (>9 years of schooling).

The age taken into consideration was based on any identification document presented, and it was divided into: 60 to 64, 65 to 74, and >75 years old.

All subjects underwent a comprehensive geriatric assessment consisting of a functional and a cognitive evaluation, as well as of an interview with an informant. The functional evaluation consisted of applying both the evaluation scale of the daily life basic activities¹⁷ and of the daily life instrumental activities¹⁸, both in the subject as well as in the informant, with divergencies in the information written down and commented by the rater. The cognitive evaluation consisted of applying the MMSE⁶, the Clock Drawing Test¹⁹, and the Verbal Fluency Test²⁰ - animal category. The Brazilian version of the Questionnaire on Cognitive Decline in the Elderly - IQCODE²¹ was applied during the interview with the informant. The diagnosis of dementia was given by the geriatrician who assisted the subject based on DSM-IV criteria.

The ICC was used to evaluate reliability, and Cronbach' alpha (α) to evaluate the test items' internal consistency, as well as the items of their subscales separately.

As the "attention" and the "calculation" subscales are very small (each having only two items), they were analyzed together with a maximum value of 9 points.

In the test-retest study, analyses were made after sample stratification by sex, age, schooling, and the presence of dementia. For all statistical analyses, a confidence interval (CI) of 95% was adopted. For the interpretation of values, the criteria proposed by Landis and Kock²² were taken into consideration: nearly perfect (>80), substantial (from 0.61 to 0.80), moderate (from 0.41 to 0.60), regular (from 0.21 to 0.40), weak (from 0.01 to 0.20), and poor (<0). The data was typed and analyzed using SPSS 12.0 for Windows.

The study was approved by the Research Ethics Committee of the Social Medicine Institute of Rio de Janeiro State University, and all participants signed an Informed Consent.

RESULTS

One hundred and twenty-three subjects took part in the test-retest study, while 60 of them participated in the interrater reliability study as well. The internal consistency of the set of items of the test was high: the Cronbach' alpha was 0.89.

Interrater reliability

In the interrater reliability study, 70% of the subjects were female; the mean ± SD value of schooling was 4.5 ± 3.9 years; mean ± SD value of age was 77.2 ± 6.9 years (Table 1). ICC was high for the set of items of the test, as well as for the items of the subscales separately and the MMSE items (>0.93). As shown in Table 2, the mean ± SD value for the total score in the test, as well as for the subscales scores, was similar between the two raters. The ICC for the set of items in the Br-CAMCOG-R was high: 0.93 (0.89-0.95) and the ICC scores for the set of items of the test as well as for the items of the subscales separately and the MMSE items were excellent (>0.92), with narrow reliability intervals.

Test-retest reliability

In the test-retest reliability study, 72.4% of the subjects were female; the mean ± SD value of schooling was 4.1 ± 4.8 years; the mean ± SD value of age was 76.1 ± 7.1 years (Table 1). In the first testing, the mean time spent was 43 ± 9.4, range 23-90 minutes; the overall mean was of 68.3 ± 12.9, range 36-96 points. In the retest, the mean time was 40.5 ± 8.9, range 24-90 minutes; the overall mean was 71.8 ± 12.6, range 42-99 points (Table 3).

The stability of the instrument with time did not vary between males and females and or with or without dementia. However, a small difference was found for subjects with low schooling (Table 4).

Discreet differences were noticed in the ICC values for the subscale items after sex, age, and schooling stratification. According to the dementia diagnosis, the "perception" and "executive functions" subscales obtained a higher ICC among the non-cases, while in the "attention and calculation" subscale, the ICC was higher among the cases (Table 4).

The subscale means were similar in the two testings; a discreet difference was noticed in the "executive functions" subscale and in the MMSE. The ICC score was excellent for the set of items of the test and for the items of the subscales; it varied from "moderate" (0.59, "praxis" subscale) up to "nearly perfect" (0.89, "memory" subscale). There was no significant range in the stability of the instrument in sex, age, schooling, or the presence of dementia. The internal consistency for the set of items in the test was 0.89 (Table 4).

DISCUSSION

Different comprehensive evaluation instruments have been developed to assess cognitive dysfunctions in the field of neuropsychology, as it is the case with the CAMCOG-R; neuropsychological test batteries are made up of instruments utilized to evaluate several cognitive functions. When a subject answers one item correctly, it is expected that he or she would also be able to answer other items related to it correctly as well, because split cognition into cognitive domains is more didactical than anatomical, since the brain operates in a network where one area is intimately connected with the other²³.

Bearing in mind the multidimensionality of the construct named cognition, the internal consistency of the instruments used to evaluate it should reflect the agreement (or not) of each item with the test as a whole.

The high score of Cronbach' alpha found in this study for the instrument' set of items suggests that the Br-CAMCOG-R evaluates one same construct. The internal consistency score of the test' items of the subscales reflected the differences of the number of items in each one of them; the subscale containing the smallest number of items obtained the lowest alpha score (attention and calculation=0.47), and those with a higher number of items obtained a higher score (memory and language=0.69). Heinik et al.¹⁶, who developed the Hebrew version of the CAMCOG and studied the internal consistency of the version in 55 subjects in a psychiatric clinic, obtained internal consistency scores varying from 0.32 (calculation subscale) to 0.93 (memory subscale); the calculation subscale has only 2 items, and perhaps, for this reason, it obtained the lowest score for internal consistency, due to the fact that it was evaluated separately, differently from our sample, where the items of the calculation subscale were added to the attention subscale.

When evaluating the stability of an instrument, it is fundamental to guarantee that the variability of the results is not significantly determined by the range associated to the rater, which can occur due to the precarious standardization while applying it. The high interrater reliability found in the present study suggests that, despite its complexity and the dependence on the raters' evaluation of some items, the variability associated to the rater did not jeopardize the instrument' reliability.

The operational difficulties during the interrater reliability study, which demanded the presence of two raters in the interview room at the same time, caused the studied sample to be smaller than that of the test-retest. However, it does not seem to us that this fact jeopardized the quality of the results, since similar studies were made with samples containing 40 to 55 subjects^8,10.

Another source of variability which may risk the stability of an instrument is the change of the studied phenomenon. The high reliability of the test-retest found in the present study suggests that the instrument has an excellent level of stability with time, for both the set of items in the test and for the majority of the items in the subscales. The data suggest that the studied phenomenon - cognition - was kept stable during the period of time in which it was studied. Tombach et al.²⁴ evaluated the test-retest reliability of the MMSE at a 90-day interval, and considered that there had not been significant changes in the subjects' cognition during that time.

The praxis subscale demonstrated the smallest stability in time (Table 4). This finding can be partially explained by the fact that the punctuation of the items which constitute the subscale depends on the raters' interpretation. It is made up of items which require the copying of drawings and the producing of a freehand drawing of a clock (without copying).

The stability of the total score of the instrument in time was adequate, even after the stratification by sex, age, schooling, and the presence of dementia; the difference in the ICC scores for the test' set of items and for most subscales was discreet, suggesting the instrument maintains its stability in time even in subjects with cognition impairment. These findings corroborate those of Lindeboom et al.⁹ which evaluated the CAMCOG' stability in elderly subjects with dementia at a 21-day interval. The stability of the instrument among those with mild, moderate and severe dementia was 0.97, 0.97, and 0.96, respectively.

The overall mean in the retest was discreetly higher than in the test, which showed a possible learning effect, already noticed in similar studies with different cognitive instruments²⁵.

A few general methodological aspects deserve comments when examining the reliability of a rating process, principally of the interrater type. One aspect has to do with the raters who, aware of the evaluation process in course, tend to proceed more rigorously than in everyday circumstances, which may provide for an overestimation of the reliability results. The other aspect is the raters' tendency to alter their approach with time, in such a way as to improve the way they ask the questions and relate to the respondents, in case the application of an instrument takes a long time²⁶.

In the interrater reliability study, the minimum age of the subjects was 65 years; for this reason, it was not possible to test the items of the remote memory subscale suggested for subjects who were born after 1940. Other study limitations were that most of the subjects had low schooling, were under 85 years old, and attended a public health clinic, making the sample a homogeneous one, but not representative of the Brazilian elderly population as a whole.

Ongoing additional criteria validity studies should complete the measurement equivalence of the Brazilian version of the test. However, we suggest reapplying this study in different operational settings to learn about the stability of the instrument in other circumstances. We reach the conclusion that the Br-CAMCOG-R has high levels of stability with time, agreement among raters, and very good internal consistency and, as such, it can be useful in epidemiological studies and in specialized clinics to evaluate cognitive functions in elders.

Received 16 December 2008.

Accepted 24 March 2009.

Dr. Emylucy Martins Paiva Paradela - Cuidado Integral à Pessoa Idosa/Policlínica Piquet Carneiro - Rua Marechal Rondon 381/2º andar - 20950-000 Rio de Janeiro RJ - Brasil. E-mail: [e-mail]

• 1. Morris JC, Heyman A, Mohs RC, et al. The Consortium to Establish a Registry for Alzheimer' disease (CERAD). Part I. Clinical and neuropsychological assessment of Alzheimer' disease. Neurology 1989;39:1159-1165.
• 2. Russel EW, Russel SLK, Hill BD. The fundamental psychometric status of neuropsychological batteries. Arch Clin Neuropsychol 2005;20:785-794.
• 3. Huppert FA, Jorm A, Brayme C, et al. Psychometric properties of the CAMCOG and its efficacy in the diagnosis of dementia. Ageing Neuropsychol Cognition 1996;3:201-214.
• 4. Williams JG, Huppert FA, Matthews FE, Nickson J. The MRC Cognitive Function and Ageing Study (MRC CFAS). Performance and normative values of a concise neuropsychological test (CAMCOG) in an elderly population sample. Int J Geriatr Psychiatry 2003;18:631-644.
• 5. Roth M, Huppert F, Mountjou CQ, Tym E. CAMDEX-R: The Cambridge examination for mental disorders of the elderly. Cambridge University Press, 1998.
• 6. Folstein MF, Folstein SE, Mc Hugh PR. Mini-mental State: a practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 1975;12:189-198.
• 7. Hodkinson M, Qureshi K. Evaluation of a 10 question mental test of the institutionalized elderly. Age Ageing 1974;3:152-157.
• 8. Roth M, Tym E, Mountjoy CO, Huppert FA, Hendrie H, Verma S, Goddard R. CAMDEX: a standardized instrument for the diagnosis of mental disorders in the elderly with special reference to the early detection of dementia. Br J Psychiatry 1986;149:698-709.
• 9. Lindeboom J, Horst RT, Hooyer C, Dinkgreve M, Jonker C. Some psychometric properties of the CAMCOG. Psychol Med 1993;23:213-219.
• 10. Heinik J, Werner P, Mendel A, Raikher B, Bleich A. The Cambridge Cognitive Examination (CAMCOG): validation of the Hebrew version in elderly demented patients. Int J Geriatr Psychiatry 1999;14:1006-1013.
• 11. Verhey FRJ, Huppert FA, Korten ECCM, et al. Cross - National Comparisons of the Cambridge Cognitive Examination- Revised: The CAMCOG-R. Results from the European Harmonization Project for instruments in dementia. Age Ageing 2003; 32:534-540.
• 12. Athey RJ, Porter RW, Walker RW. Cognitive assessment of a representative community population with Parkinson' disease (PD) using the Cambridge Cognitive Assessment-Revised (CAMCOG-R). Age Ageing 2005;34:268-273.
• 13. Athey RJ, Walker RW. Demonstration of cognitive decline in Parkinson' disease using the Cambridge Cognitive Assessment (Revised) (CAMCOG-R). Int J Geriatr Psychiatry 2006; 21:977-982.
• 14. Lautenschlager NT, Dunn JC, Bonney K, Flicker L, Almeida OP. Latent Semantic Analysis: an improved method to measure cognitive performance in subjects of Non-English-Speaking-Background. J Clin Exp Neuropsychol 2006;28:1381-1387.
• 15. Heinik J, Solomesh I. Validity of the Cambridge Cognitive Examination-Revised new Executive Function Scores in the diagnosis of dementia: some early findings. J Geriatr Psychiatry Neurol 2007;20:22-28.
• 16. Bottino CMC, Stoppe AJr, Scalco AZ, Ferreira RCR, Hototian SR, Scalco MZ. Validade e confiabilidade da versão brasileira do CAMDEX. Arq Neuropsiquiatr 2001;59(Suppl 3):S20.
• 17. Lino VTS. Adaptação transcultural da Escala de Independência em Atividades da Vida Diária (Escala de Katz). Cad Saúde Pública 2008;24:103-112.
• 18. Lawton MP, Brody EM. Assessment of older people: self maintaining and instrumental activities of daily living. Gerontologist 1969;9:179-186.
• 19. Manos PJ, Wu R. The ten point clock test: a quick screen and grading method for cognitive impairment in medical and surgical patients. Int J Psychiatry Med 1994;24:229-244.
• 20. Bertolucci PHF, Okamoto IH, Brucki SMD, Siviero MO, Toniolo J Neto, Ramos LR. Applicability of the CERAD Neuropsychological Battery to Brazilian Elderly. Arq Neuropsiquiatr 2001;59:532-536.
• 21. Sanchez MAS. Questionário baseado no relato do informante para detecção de declínio cognitivo em idosos: tradução, adaptação transcultural e estudo da confiabilidade. Dissertação. Rio de Janeiro, 2007.
• 22. Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-174.
• 23. Mungas D, Reed BR, Farias ST, Decarli C. Criterion-referenced validity of a neuropsychological test battery: equivalent performance in elderly Hispanics and Non-Hispanic whites. J Intern Neuropsychol Soc 2005;11:620-630.
• 24. Tombaugh TN Test-retest reliable coefficients and 5-year change scores for the MMSE and 3MS. Arch Clin Neuropsychol 2005;20:485-503.
• 25. De Yebenes MJ, Otero A, Zunzunegui MV, Rodriguez-Laso A, Sanchez-Sanchez F, Del Ser T. Validation of a short cognitive tool for the screening of dementia in elderly people with low educational level. Int J Geriatr Psychiatry 2003;18:925-936.
• 26. Hasselmann MH, Lopes CS, Reichenheim ME. Confiabilidade das aferições de estudo sobre violência familiar e desnutrição severa na infância. Rev Saúde Pública 1998;32:437-446.

Publication Dates

• Publication in this collection
  14 July 2009
• Date of issue
  June 2009

History

• Accepted
  24 Mar 2009
• Received
  16 Dec 2008