Translation and validation of the Brazilian version of the Cerebral Palsy Quality of Life Questionnaire for Children – child report^☆

Abstract

Objective:

To verify the psychometric properties of the Cerebral Palsy: Quality of Life Questionnaire Children – child report (CPQol-Child) questionnaire, after it was translated and culturally adapted into Brazilian Portuguese.

Methods:

After the translation and cultural adaptation of the tool into Brazilian Portuguese, the questionnaire was answered by 65 children with cerebral palsy, aged 9–12 years. The intraclass correlation coefficient and Cronbach's alpha were used to assess the reliability and internal consistency of the tool and its validity was analyzed through the association between CPQol-Child: self-report tool and Kidscreen-10 using Pearson's correlation coefficient.

Results:

Internal consistency ranged from 0.6579 to 0.8861, the intraobserver reliability from 0.405 to 0.894, and the interobserver from 0.537 to 0.937. There was a weak correlation between the participation domain and physical health of CPQol-Child: self-report tool and Kidscreen-10.

Conclusion:

The analysis suggests that the tool has psychometric acceptability for the Brazilian population.

Keywords
Quality of life; Cerebral palsy; Child

Resumo

Objetivo:

Verificar as propriedades psicométricas da versão traduzida e adaptada culturalmente para o português do Brasil do instrumento Cerebral Palsy Quality of Life Questionnaire for Children – Child report.

Métodos:

Após a tradução e a adaptação cultural do instrumento para o português, o questionário foi respondido por 65 crianças com paralisia cerebral, entre nove e 12 anos. Os coeficientes de correlação intraclasse e alfa de Cronbach foram usados para avaliar a confiabilidade e consistência interna do instrumento e a validade do instrumento foi analisada pela relação entre CPQol-Child: self-report toole a Kidscreen-10 por meio do coeficiente de correlação de Pearson.

Resultados:

A consistência interna variou de 0,6579 a 0,8861, a confiabilidade intraobservador de 0,405 a 0,894 e a interobservador de 0,537 a 0,937. Verificou-se uma fraca correlação entre o domínio participação e saúde física da CPQol-Child e Kidscreen-10.

Conclusão:

A análise feita sugere que o instrumento usado tem aceitabilidade psicométrica para a população brasileira.

Palavras-chave
Qualidade de vida; Paralisia Cerebral; Criança

Introduction

Cerebral palsy (CP) is a group of movement and posture disorders that causes limitations in activities due to non-progressive alterations that occur in the fetal or infant brain, usually accompanied by sensory, cognitive, communication, perception, behavior alterations, and/or seizures.¹ It is the most common cause of motor disability in childhood,² with an incidence in developed countries of 2–2.5/1000 live births.³ Although there is no accurate data in Brazil, some authors estimate an incidence of 7/1000 live births.⁴

The need to know the effects of the disease on the health and well-being has resulted in several efforts to develop tools to assess the quality of life (QOL) of these children,⁵ mainly tools in which the respondent is the child itself, as there appears to be a discrepancy between self-report of children and adolescents and their caregivers, especially regarding the emotional aspects.^6–8 There is evidence that children can reliably answer the QOL self-report if their emotional development, cognitive capacity, and reading level are considered⁹; however, one must be cautious about the reliability of information provided by very young children, as well as by those with cognitive deficit or severe communication impairment.

Currently, there are generic tools to assess QOL of children that have been translated and validated for the Portuguese language, but they do not address specific characteristics of CP. A specific tool must be used in CP, i.e., one that addresses the feelings on assistive technology equipment; feelings about medical, therapeutic and surgical interventions; satisfaction with access to services; and acceptance in the community. These issues go beyond the scope of a generic tool, which usually omits information on the daily lives of these children and do not address the point of view of children with CP, generating doubts whether they correspond to their opinion.^10–12 A study carried out in 2007¹¹ identified only two specific tools to assess QOL of children with CP; the DISABKIDS-Cerebral palsy and the Cerebral Palsy: Quality of Life Questionnaire Children (CPQol-Child).¹²

The CPQol-Child: self-report tool is considered as a tool with strong psychometric properties to assess QOL in these schoolchildren,¹⁰ and has been widely used.

The authors of the CPQol-Child: self-report tool developed a tool based on the International Classification of Functioning, Disability and Health (ICF), with the help of a team of international experts, and took into account the views of the child and caregivers. The questionnaire has two versions, the CPQol-Child: self-report tool Primary Caregiver Questionnaire (4–12 years) and CPQol-Child: Child Report Questionnaire (9–12 years). The CPQol-Child: Child Report Questionnaire (9–12 years) version is answered by children with CP aged 9–12 years and contains 53 questions distributed in the following domains: social well-being and acceptance, functionality, participation and physical health, emotional well-being and self-esteem, access to services, and pain and impact of the disability.^12,13 The version for caregivers has already been translated into different languages.^14–17 The World Health Organization (WHO) recommends the translation and cultural adaptation of existing tools because it facilitates the comparison of studies conducted in different countries and the communication between researchers.¹⁸

From this perspective, this study aimed to determine the psychometric properties of the translated and culturally adapted version for the Brazilian Portuguese of the tool CPQol-Child: Child Report Questionnaire – 9–12 years.

Materials and methods

The translation, cultural adaptation, and validation into Brazilian Portuguese of the CPQol-Child: self-report tool was requested and authorized by its authors.The project was approved by the Research Ethics Committee of Faculdade de Filosofia e Ciências, under N. 278/2009.

Participants

To calculate the minimum sample size, a correlation coefficient equal to or greater than 0.40 was considered for the validity; for reproducibility, an intraclass correlation coefficient (ICC) equal to or greater than 0.40, a type I error of 5%, and a type II error of 20%, with a 30% increase for possible losses or refusals, resulting in a minimal sample size of n = 62.¹⁹

The initial sample consisted of 200 patients with CP, from different regions of the country, and after verifying the inclusion and exclusion criteria, 65 children were selected. The inclusion criteria were age between 9 and 12 years and CP diagnosis. The exclusion criteria were any intellectual deficits and lack of an efficient communication system.

The study used a convenience sample due to the access difficulties of the participants with the necessary characteristics. According to Mattar,²⁰ a non-probabilistic sample is a viable alternative when the population is not available to be drawn by lots and when there are time, financial, and material resource limitations. The parents/guardians signed an informed consent form and a consent term was read to the children. The consent term was read, explained, and the child was asked for voluntary participation. The agreement was obtained verbally or non-verbally due to the motor difficulties shown by many of the participants, which prevented the signing of the document.

Procedures

The process of translation and adaptation followed international recommendations,^21–23 and the following steps were carried out: translation into Portuguese; reconciled translation; back translation; final translation; pretest; cultural adaptation.

Firstly, two translators independently translated the CPQol-Child: self-report tool from English into Portuguese. Both had fluency in English and Portuguese, and Portuguese as their native language. The following guidelines were given to the translators: use a natural and acceptable language for a broad audience; make a clear, simple and understandable translation; avoid long sentences; focus on the conceptual equivalence rather than literal translation; consider the age of the respondents and how they will understand the items; do not use slang or terms that are difficult to understand; avoid using double negatives.²¹

Subsequently, both translations were compared resulting in a reconciled translation that consisted of a consensual version with item adequacy and reconciliation. At this stage, the collaboration of a team of researchers that had experience with CP children was requested. They had to analyze item by item, choose the best translation, and suggest another translation if necessary. They were asked to focus on the cultural and linguistic differences that could cause difficulties when the English version was translated into Portuguese.

Then, a native English translator fluent in Portuguese was asked to carry out the back-translation, that is, translate the reconciled translation into the English language, which was sent to the authors of the original questionnaire to identify and correct the discrepancies regarding the semantic, idiomatic, and conceptual equivalence.

The fourth step consisted in the review and comparison of the back-translated version corrected by the questionnaire authors with the original version in English that generated the final translation, which was used in the pretest.

In the pretest, the final version was applied to six children with CP to verify whether all items were comprehensible and satisfactory. In order to test the cultural equivalence, the questions that did not lead to a good understanding were discussed once again, reformulated by the researchers, and applied to another group of six children, until all items of the questionnaire were understood by 90% of the respondents.²⁴

After completion of this stage, an invitation was sent to researchers from Brazilian universities who work with children with CP to assist in the study. After the acceptance, meetings were held for training of the theoretical and methodological aspects of questionnaire application. At the training, the interviewers were instructed to read each question to the child and request an answer, which was noted in the questionnaire. They could not make any intervention or comment on the question or the answer. The interviews were carried out by two researchers in each region of the country. As there was no intervention, explanation, or comments during the interviews, the differences between evaluators did not influence the answers.

For data collection, a mapping of the number of children diagnosed with CP and the age range of these children was performed. Two hundred children were identified that met the established criteria and 65 families agreed that their children's participation in the study.

During the collection, each child was interviewed three times by two different interviewers: I1 and I2. Interviewer I1 carried out the first interview, when the Portuguese version of the CPQol-Child: self-report tool was answered (Fig. 1) [online only], and filled out a questionnaire with information on the child's gender, age, level of education, and classification according to the Gross Motor Function Classification System (GMFCS). After an interval of 30–60 min, interviewer I2 performed the second interview, in which the child once again answered the Portuguese version of the CPQol-Child. After 14 days, a third interview was carried out by interviewer I1, in which the participants answered the Portuguese versions of CPQol-Child: self-report tool and Kidscreen-10.

The translated and validated version for Brazilian Portuguese of the Kidscreen-10 questionnaire was used to verify construct validity. This tool was chosen because it has been shown to be effective for the generic assessment of QOL in healthy children, as well as those with chronic conditions. This tool has international quality standard and was used by the authors of the CPQol-Child: self-report tool for validation of the original tool in English and has been used in the translations to the different languages and cultures.

The GMFCS is a classification system of the motor function level of children with CP that has been used internationally and allows for stratification into five skill levels. Level I represents the best gross motor skills and level V, the worst function, based on the age of the assessed child.²⁵

The typing of raw data was carried out using SPSS (IBM Corp. Released 2013; IBM SPSS Statistics for Windows, version 22.0, NY, USA) to verify the score of each questionnaire and domain for future statistical analysis.

Statistical analysis

Descriptive statistics to characterize the study population were performed, as well as tests to verify the inter- and intra-observer reliability and internal consistency of the tool.

The interobserver reliability was evaluated based on measurements made at the same time by different interviewers, and the intraobserver reliability was assessed through test and retest, which consisted in completing the questionnaire twice, with enough time between them to exclude the memory effect, but not long enough to avoid changes in QOL.¹¹ The internal consistency of the tool was also assessed, which consists in verifying whether the repeated measures within the same scale are convergent, meaning that they are directed at the same direction, whether the items in each dimension form a coherent whole, and whether the approximate internal correlation between items is relatively strong.¹¹

The ICC and Cronbach's alpha were used to assess the tool's reliability and internal consistency.^12,13 The ICC was considered excellent when ICC ≥ 0.75; satisfactory when 0.4 ≤ ICC < 0.75 and weak when ICC < 0.4¹⁹; p-values <0.05 were considered significant. As for the Cronbach's alpha, measures with a reliability >0.5 are recommended to compare groups of individuals.²⁶ The construct validity was analyzed using Pearson's correlation coefficient.

Results

Table 1 shows the data related to the demographic characteristics of the children with CP. There was a predominance of males with GMFCS level I, although children from all strata were interviewed.

Table 1
Children's demographic characteristics.

Variable		n (%)
Age (years) mean ± SD		10.5 ± 1.25
Gender
	Male	41 (63)
	Female	24 (37)
GMFCS level
	I	27 (41)
	II	18 (27)
	III	7 (11)
	IV	5 (7)
	V	9 (14)
Region of Brazil
	Southeast	46 (70)
	South	04 (06)
	Midwest	05 (08)
	Northeast	07 (11)
	North	03 (05)

The reliability of CPQol-Child: self-report tool questionnaire was adequate, with Cronbach's alpha coefficient >0.5 for all domains, for both evaluators. Only the domain "access to services" showed values <0.7 (Table 2).

Table 2
Internal consistency of the Brazilian Portuguese version of the Cerebral Palsy Quality of Life Questionnaire for Children – child report.

CP QOL child subscale	Cronbach's	Cronbach's
	alpha	alpha
	E 1	E 2
Social well-being and acceptance (12 items)	0.8050	0.8490
Functionality (12 items)	0.8861	0.8618
Participation and physical health (11 items)	0.7975	0.7873
Emotional well-being and self-esteem (6 items)	0.8445	0.8445
Access to services (3 items)	0.6579	0.6579
Pain and disability impact (8 items)	0.7758	0.7416

Table 3 shows the results of intra- and interobserver reliability for each CPQol-Child: self-report tool domain. The intraobserver reliability was significant for all domains; it was considered excellent for functionality (ICC ≥ 0.75), and adequate for social well-being and acceptance, physical health participation, emotional well-being and self-esteem, access to services, and pain and impact of disability (0.4 ≤ ICC < 0.75). The interobserver reliability was significant for all domains; it was considered excellent for emotional well-being and self-esteem, as well as for access to services (ICC ≥ 0.75), and satisfactory for social well-being and acceptance, functionality, participation and physical health, and pain and impact of the disability (0.4 ≤ ICC < 0.75).

Table 3
Intra- and interobserver reliability of each domain of the Portuguese version of the Cerebral Palsy Quality of Life Questionnaire for Children – child report evaluated by the intraclass correlation coefficient.

Domain	Intraclass correlation coefficient (ICC)
	Intra-observer (test–retest)	Inter-observer
Social well-being and acceptance	0.732	0.625
Functionality	0.894	0.725
Participation and physical health	0.640	0.537
Emotional well-being and self-esteem	0.600	0.848
Access to services	0.562	0.937
Pain and disability impact	0.405	0.675

Table 4 shows a weak correlation between the domain participation and physical health of CPQol-Child: self-report tool and Kidscreen-10.

Table 4
Correlation between the domains of the Cerebral Palsy Quality of Life Questionnaire for Children – child report and Kidscreen 10.

CP QOL-Child
	SWA	FU	PPH	EWB	AS	PI
Kidscreen 10	0.216	0.164	0.277	0.190	−0.056	0.057
	0.128	0.250	0.049	0.182	0.694	0.690

• SWA, social well-being and acceptance; FU, functionality; PPH, participation and physical health; EWB, emotional well-being and self-esteem; AS, access to services; PI, pain and disability impact.

Discussion

The study allowed for the creation of the Portuguese version of the CPQol-Child: self-report tool questionnaire, a specific tool that assesses QOL of children with CP, which will allow measuring and comparing the QOL of Brazilian children based on the self-report and might contribute to establish public policy parameters in health and education, and also verify the effectiveness of the developed therapeutic and preventive actions applied to this population.

The methodology used followed experts' recommendations^22,23 and ensured an appropriate version of the tool regarding the cultural aspects of the Brazilian population, and equivalent to the original version in English.

Regarding the characteristics of the study participants, the authors aimed to ensure the representativeness of the population and the different types of CP, although there was a predominance of participants from the Southeast region. It is noteworthy that the study included participants from different regions of the country to ensure that the multicultural characteristics of Brazil did not result in difficulties to understand some of the questions and provide the answers.¹⁷

The study results indicated that the Portuguese version of CPQol-Child: self-report had reliability and validity to assess the QOL of Brazilian children with CP aged between 9 and 12 years.

Several authors have emphasized the importance of validating a questionnaire for children through self-report, as children and adolescents have different degrees of perception of themselves and the world, and therefore, a different view of their QOL. The perception of adults, even those who live in close contact with the children, has in general a low degree of correlation with the child's self assessment.^6,7,27 It should be noted that factors such as age, severity of motor impairment, and functionality affect the QOL of individuals with CP and, with increasing age, the level of participation of these children in activities decreases at the same time as their capacity for reflection increases, which interferes with their perception of QOL.²⁸

The data indicated a high degree of internal consistency, which are similar results to those found in the original version of CPQol-Child: self-report.¹³

The lowest value obtained in ICC for test–retest reliability (0.405) was for the domain "pain and impact of disability"; however, the interobserver ICC for this domain had a higher value (0.675), which may indicate the fact that the question is directed to a subjective state that could change during the 14-day period, the time required between the first and second assessments. The domain "emotional well-being and self-esteem", which had a value of 0.600 in the intraobserver assessment, also refers to the subjective evaluation that can exhibit changes in a short period of time. However, the value obtained for the domain "access to services" in the intra-observer assessment may indicate that there was a difficulty in understanding this domain by the study participants.

Regarding construct validity, the weak correlation found for the domain "participation and physical health" differs from the results by Davis et al.,⁵ who found a moderate correlation. However, as stated by those authors, there are conceptual differences between these tools, commonly used to assess QOL of children with CP, which may have affected the results. The CPQol-Child: self-report tool is a specific tool to assess QOL of children with CP and aims to understand how the child feels about aspects of his/her life regarding different domains, while the Kidscreen-10 is a generic tool on health-related children's QOL that is easy to be applied, but offers only a summarized score. The health-related QOL assessment using a single value can miss information related to some physical and psychosocial aspects.²⁹

The study had limitations, especially regarding the lack of additional data on the socio-economic characteristics, the heterogeneous distribution of the sample between the regions, and the prevalence of children with levels I and II of GMFCS. There was a greater sample loss in some regions of the country, as they did not meet the inclusion criteria. The prevalence of children at these levels occurred because children with more severe CP often have associated comorbidities, such as intellectual deficit and severe communication problems, which prevented their participation in the study. However, it is believed that the questionnaire can be used safely and reliably in any region of the country, as long as the respondents meet the criteria established in the user's manual.

The Brazilian version of the CPQol-Child: self-report tool showed adequate psychometric properties and is a reliable tool, easy to understand, and easily applicable to evaluate the QOL of Brazilian's children with CP through self-report.

The access to the questionnaire can be attained by registering at the website http://www.cpqol.org.au/questionnaires_manuals.html.

• ☆
  Please cite this article as: Braccialli LM, Almeida VS, Sankako AN, Silva MZ, Braccialli AC, Carvalho SM, et al. Translation and validation of the Brazilian version of the Cerebral Palsy Quality of Life Questionnaire for Children - child report. J Pediatr (Rio J). 2016;92:143-8.
• Funding
  This work was funded by CNPq.

Acknowledgements

The authors would like to thank Elizabeth Waters and her team for permission to translate the tool. They also thank CNPq for the financial support.

References

• ¹
  BaxM, Goldstein M, Rosenbaum P. Leviton A, Paneth N. Proposed definition and classification of cerebral palsy, April 2005. Execu-tive Committee for the Definition of Cerebral Palsy. Neurology. 2005;47:571-6.
• ²
  Msal ME, Park JJ. Neurodevelopmental management strategies for children with cerebral palsy: optimizing function, promo-ting participation, and supporting families. Clin Obstet Gynecol. 2008;51:800-15.
• ³
  Surveillance of Cerebral Palsy in Europe. Surveillance of cerebral palsy in Europe (SCPE): a collaboration of cerebral palsy surveys and registers. Dev Med Child Neurol. 2000;42:816-24.
• ⁴
  Lima CL, Fonseca LF. Paralisia cerebral: neurologia, ortopedia e reabilitação. Rio de Janeiro: Guanabara Koogan; 2004.
• ⁵
  Davis E, Shelly A, Waters E, Davern M. Measuring the quality of life of children with cerebral palsy: comparing the conceptual differences and psychometric properties of three instruments. Dev Med Child Neurol. 2010;52:174-80.
• ⁶
  Prebianchi H. Medidas de qualidade de vida para crianças: aspectos conceituais e metodológicos. Psicol.: Teor. Prát. 2003;5:57-69.
• ⁷
  Eiser C. Children's quality of life measures. Arch Dis Child. 1997;77:350-4.
• ⁸
  Varni J, BurwinkleT Sherman S. Health-related quality of life of children and adolescents with cerebral palsy: hearing the voices of the children. Dev Med Child Neurol. 2005;47:592 -7.
• ⁹
  Dickinson HO, Parkinson KN, Ravens-Sieberer U, Schirripa G, Thyen U, Arnaud C, et al. Self-reported quality of life of 8-12-year-old children with cerebral palsy: a cross-sectional European study. Lancet. 2007;369:2171-8.
• ¹⁰
  Carlon S, Shields N, Yong K, Gilmore R, Sakzewski L, Boyd R. A systematic review of the psychometric properties of quality of life measures for school aged children with cerebral palsy. BMC pediatrics. 2010;10:81.
• ¹¹
  ViehwegerE, RobitailS, RohonMA, JacquemierM, JouveJL, Bol-lini G, et al. Measuring quality of life in cerebral palsy children. Ann Readapt Med Phys. 2008;51:129-37.
• ¹²
  Waters E, Maher E, Salmon L, Reddihough D, Boyd R. Deve-lopment of a condition-specific measure of quality of life fo children with cerebral palsy: empirical thematic data reported by parents and children. Child Care Health Dev. 2005;31:127-35.
• ¹³
  Waters E, Davis E, Mackinnon A. Psychometric properties of the quality of life questionnaire for children with CP. Dev Med Child Neurol. 2007;49:49-55.
• ¹⁴
  Wang HY, Cheng CC, Hung JW, Ju YH, Lin JH, LoSK. Lo Validating the Cerebral Palsy Quality of Life for Children (CP QOL-Child) questionnaire for use in Chinese populations. NeuropsycholRehabil. 2010;20:883-98.
• ¹⁵
  Akbarfahimi N, Rassafiani M, Soleimani F. Vameghi R, Kazem-nejad A, Nobakht Z. Validity and reliability of farsi version of Cerebral Palsy-Quality Of Life Questionnaire. J Rehabil Med. 2013;13:73-83.
• ¹⁶
  Dmitruk E, Mirska A, Kutak W, Kalinowska AK, Okulczyk K, Wojt-kowski J. Psychometric properties and validation of the Polish CP QOL-Child questionnaire: a pilot study. Scand J Caring Sci. 2014;28:878-84.
• ¹⁷
  Braccialli LM, Braccialli AC, Sankako AN, Dechandt ML, Almeida VS, Carvalho SM. Questionário de qualidade de vida de criancças com paralisia cerebral (CpQol-Child): traducão e adaptacão para língua portuguesa. J. Hum. Growth Dev. 2013;23:1-10.
• ¹⁸
  World Health Organization. The World Health Organization Quality of Life Assessment (WHOQOL): position paper from the World Health Organization. SocSci Med. 1995;41:1403-9.
• ¹⁹
  Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N, Newman TB. Designing clinical research: an epidemiologic approach. Philadelphia, PA: Lippincott Williams & Wilkins; 2001.
• ²⁰
  Mattar FN. Pesquisa de marketing: metodologia, planejamento. 5 ed. São Paulo: Atlas, 1999.
• ²¹
  Waters E, Davis E, Boyd R, Reddihough D, Mackinnon A, Graham HK, et al. Cerebral palsy quality of life questionnaire for children (CP QOL- Child) Manual. In: University D, (ed.), Children. Melbourne: Deakin University; 2006. 1-31.
• ²²
  Waters E, Davis E, Boyd R, Reddihough D, Mackinnon A, Graham HK, et al. Cerebral palsy quality of life for children translation guidelines. Melbourne: Deakin University; 2006. p. 16.
• ²³
  Guillemin F. Cross-cultural adaptation and validation of health status measures. Scand J Rheumatol. 1995;24:61-3.
• ²⁴
  de Soárez PC, Kowalski CC, Ferraz MB, Ciconelli RM. Tradução para português brasileiro e validacão de um questionário de avaliaçcão de produtividade. Rev Panam Salud Publica. 2007;22:21-8.
• ²⁵
  Gorter JW, Rosenbaum PL, Hanna SE, Palisano RJ, Bartlett DJ, Russell DJ, et al. Limb distribution, motor impairment, and functional classification of cerebral palsy. Dev Med Child Neurol. 2004;46:461-7.
• ²⁶
  Maroco J, Garcia-Marques T. Qual a fiabilidade do alfa de Cron-bach? Questõesantigas e soluções modernas. Laboratório de Psicologia. 2006;4:65-90.
• ²⁷
  Mc Manus V, Corcoran P Perry IJ. Participation in everyday acti-vities and quality of life in pre-teenage children living with cerebral palsy in South West Ireland. BMC Pediatr. 2008;8:50-9.
• ²⁸
  Shikako-Thomas K, Lach L, Majnemer A, Nimigon J, Cameron K, Shevell M. Quality of life from the perspective of adolescents with cerebral palsy: "I justthink I'm a normal kid. Ijust happen to have a disability''. Qual Life Res. 2009;18:825-32.
• ²⁹
  Ravens-Sieberer U, Erhart M, Rajmil L, Herdman M, Auquier P, Bruil J, et al. Reliability, construct and criterion validity of the KIDSCREEN-10 score: a short measure for children and adolescents' well-being and health-related quality of life. Qual Life Res. 2010;19:1487-500.

Appendix A. Supplementary data

Supplementary data associated with this article can be found, in the online version, at doi: http://dx.doi.org/10.1016/j.jpedp.2015.12.005.

Publication Dates

• Publication in this collection
  Mar-Apr 2016

History

• Received
  3 Nov 2014
• Accepted
  27 May 2015