Translation, linguistic and cultural adaptation, reliability and validity of the Radboud Oral Motor Inventory for Parkinson's Disease – ROMP questionnaire

Tradução, adaptação linguística e cultural, confiabilidade e validade do questionário “Radboud Inventário Motor Oral para doença de Parkinson – ROMP”

ABSTRACT

Objective:

To translate and linguistically and culturally adapt to Brazilian Portuguese, and verify the reliability and validity of the Radboud Oral Motor Inventory for Parkinson's Disease (ROMP).

Methods:

The ROMP was translated and retranslated, and the instrument reliability was verified by analyzing the internal consistency and the reproducibility of the intra-examiner retest. The final version was applied to 27 participants with Parkinson's disease.

Results:

Internal consistency was 0.99 for the total ROMP and 0.96 to 0.99 for the three domains. Intraclass correlation coefficients for reproducibility were 0.99 for the total ROMP and 0.93 to 0.99 for the subscales. The ROMP and its subscales correlated substantially with the Likert-type scale, as well as with the unified Parkinson's disease rating scale II and III items.

Conclusion:

The linguistic and cultural equivalence of the ROMP in Brazilian Portuguese is now available, with excellent reliability and validity.

Keywords:
swallowing disorders; dysarthria; speech; Parkinson's disease; surveys and questionnaires; rehabilitation; sialorrhea

RESUMO

Objetivo:

Traduzir e adaptar linguística e culturalmente para o português brasileiro, verificar a confiabilidade e a validade do Radboud Inventário Motor Oral para Doença de Parkinson (ROMP).

Métodos:

O ROMP foi traduzido e retraduzido, e a confiabilidade do instrumento foi verificada através da análise da consistência interna e da reprodutibilidade do reteste intra-examinador, sendo a versão final aplicada em 27 participantes com doença de Parkinson (DP).

Resultados:

A consistência interna foi de 0,99 para o ROMP total e de 0,96 a 0,99 para os 3 domínios. Os coeficientes de correlação intra-classe para reprodutibilidade foram 0,99 para o ROMP total e 0,93 a 0,99 para as subescalas. O ROMP e suas subescalas correlacionaram-se substancialmente com a escala do tipo Likert, bem como com os itens UPDRS II e III.

Conclusão:

A equivalência linguística e cultural do ROMP no português brasileiro está agora disponível, com excelente confiabilidade e validade.

Palavras-chave:
transtornos de deglutição; disartria; fala; doença de Parkinson; inquéritos e questionários; reabilitação; sialorreia

Parkinson's disease (PD) is a chronic and progressive condition of the nervous system, characterized by the cardinal signs of rigidity, bradykinesia, tremor and postural instability¹. Other clinical data of importance are: gait disturbances, mask facies, dysarthria, drooling, sexual dysfunction, cramps, pain, paresthesia, dysphagia, urinary incontinence, intestinal constipation, writing disorders (micrography), sleep disturbances, bradyphrenia, depression, and dementia².

Studies show that up to 90% of people with PD may present with speech abnormalities, including defects that encompass the resonance, breathing, voice, and articulation systems, considered to originate from motor planning damage and difficulties in the execution of simultaneous or sequential characteristics of dysfunctions of the nuclei of the base^3,4. The prevalence of dysphagia in PD patients ranges from 35-100% depending on the assessment, and subjective dysphagia occurs in one-third of community-dwelling PD patients. Objectively measured, dysphagia rates were much higher, with four out of five patients being affected. This suggests that dysphagia is common in PD, but patients do not always report swallowing difficulties unless asked, and dysphagia in the first five years of the disease is considered a red flag for the diagnosis of PD^{5,6,7,8,9,10,11}. Drooling is reported by about a quarter of the population, depending on the diagnostic criteria¹².

It is important to highlight the value of instruments to identify and quantify dysarthria, dysphagia, and drooling in patients with neurodegenerative diseases. These questionnaires will allow adequate referrals to be made when the symptom is still in its initial stages, minimizing the negative impact on the individual and, consequently, improving the quality of life of these patients.

Few questionnaires are validated to self-assess speech changes, swallowing disorders, and saliva disorders in PD patients and existing surveys are lengthy or do not sufficiently specify all changes, such as the Swal-QOL for the subjective assessment of dysphagia^13,14.

Also, the few validated instruments are only available in their native language, mostly English. Of the tools researched and found in the present review, only one, the Radboud Oral Motor Inventory for Parkinson's disease (ROMP) refers to the population with PD, which is published in the English translation of its original Dutch form¹².

The ROMP was conceptually developed for the self-assessment of speech, swallowing and saliva control in patients with PD and atypical parkinsonism. In its original form, it is a valid and reliable questionnaire for clinical and scientific use, providing useful information for clinical evaluation in this population. The authors of the questionnaire have reported that the influence of cognitive deterioration on speech skills or the consequences of dysphagia was not included in generic questionnaires^15,16. Therefore, they developed this survey to evaluate the three domains of speech, swallowing, and control of saliva. They created the ROMP in such way that it detected the symptoms at the levels of their functioning and activities, according to the International Classification of Functioning, Disability, and Health, while limiting the number of items. To compensate for cognitive problems, they constructed the responses in a way that the core of each item was repeated in the possibility of response¹⁴.

Therefore, with the authorization of the authors, this research aimed to carry out the translation, linguistic and cultural adaptation, and reliability of the Brazilian version of the ROMP. In addition to validating the ROMP and its subscales against a Likert-type scale, the subscales of speech, salivation and swallowing of the Unified Parkinson's Disease Rating Scale (UPDRS) II and the speech subscale of the UPDRS III.

METHODS

The research was approved by the Research Ethics Committee of the Hospital das Clínicas of Porto Alegre under number 150339, and the individuals who agreed to participate in the study signed the informed consent form.

The ROMP is a self-evaluative protocol in three domains: speech, swallowing and saliva. It consists of 23 items, seven items in the speech domain, seven items for swallowing and nine items relating to saliva. The patient marks the frequency of symptoms from 1-5 (1 = normal; 5 = worst score). The minimum total score is 23, and the maximum score is 115 points.

Content validation (translation, linguistic and cultural adaptation), as well as the ROMP reliability, were carried out in the following steps:

1. Translation and linguistic and cultural adaptation: the ROMP was translated, adapted and validated in its content, according to Beaton et al.¹⁷. Initially, it was translated from English into Brazilian Portuguese by two Brazilian professionals fluent in both languages, one of whom had specific knowledge and the other without specific knowledge of the project. Each of the translators performed the translation separately. Afterward, they analyzed any small discrepancies and, in consensus, the English version of the questionnaire was produced. Subsequently, back-translation to Brazilian Portuguese was done by two different English translators, without specific knowledge, also fluent in both languages, resulting in two versions of back translation. Both translations were analyzed by a committee of five specialists, three speech therapists and two neurologists, who evaluated the semantic, idiomatic, experimental and conceptual equivalence, resulting in the final version.

2. Protocol application: the final protocol was then applied to 27 patients with idiopathic PD, who underwent clinical follow-up at the Movement Disorders outpatient clinic of the Hospital das Clínicas of Porto Alegre, RS, Brazil, between August 2015 and June 2016. Patients with a medical diagnosis of idiopathic PD according to the criteria of the London Brain Bank¹⁸, were at stage 2 or 3 on the Hoehn & Yahr (H&Y) scale¹⁹, and were using medication for the disease. Patients who had altered oral comprehension, auditory or visual impairment that made it impossible to perform the tasks, those who were off in relation to the medication at the time of the evaluation and those who refused to participate in the study were excluded. The need for a sample of 21 patients was initially estimated to detect a difference of 0.5 effect size, with a power of 80% and a level of significance of 5%²⁰.

3. Reliability: quantifies the internal consistency between the items of the protocol and the intra-examiner reproducibility. Cronbach's alpha (α) was used to quantify the internal consistency. Intra-examiner reliability is the measure of reproducibility of the evaluation by the same evaluator, during the repetition of the evaluation (test-retest)^21,22. The retest was applied after two weeks to reduce the chances of modification of the questionnaire responses, due to possible changes in the oral motor symptoms and to prevent the individual from remembering the protocol questions²³. The reproducibility was calculated with intraclass correlation coefficients (ICC).

4. Validity: correlation between the ROMP and a Likert-type scale. To verify this correlation, participants were asked to classify their speech, swallowing and saliva using a five-point Likert-type scale (self-perceptive): very good, good, reasonable, poor, very bad. Data were compared by Spearman's correlation (r). It is worth mentioning that the self-perception analysis is a strategy commonly used to validate self-assessment protocols in general^24,25,26,27. Correlation between ROMP and UPDRS II and UPDRS III²⁸. To verify this correlation, ROMP was compared with the subscales speech, salivation and swallowing of the UPDRS II and the subscale speech of the UPDRS III. Data were compared by Spearman's correlation. The qualified professional applying the UPDRS was blinded for the scores of the ROMP and the scale was performed on the same day.

Statistics

The internal consistency was calculated with Cronbach's α where we accepted 0.70 as a minimum for sufficient consistency²⁹. The test-retest reliability was calculated with ICCs and the correlation index 0.75 was considered the minimum acceptable agreement, since values from 0.90 were considered high³⁰. The validity of the ROMP against concurrent measures was expressed using Spearman's correlation. The level of significance adopted was p ≤ 0.01 and the analyses were performed in the Statistical Package for Social Sciences version 20.0.

RESULTS

Demographic and clinical characteristics of patients with PD

Twenty-seven patients with PD (59.3% women) were evaluated. Data on the demographic variables of the participants are shown in Table 1.

Table 1
Demographic and clinical characteristics.

Variables	Mean (SD)
Age (years)	64.9 (± 11.7)
Schooling (years)	6.6 (± 5.1)
Disease duration (years)	8.8 (± 7.1)

• SD: standard deviation.

Final version of the ROMP

The final composition of the ROMP (Figure), after translation, linguistic and cultural adaptation, has 23 self-assessment questions, seven of which are questions on the speech domain, seven questions are about the swallowing domain and nine questions are in the saliva domain. The patient marks the frequency of symptoms from 1-5 (1 = normal; 5 = worst score), as in the original.

Figure
Radboud Oral Motor Inventory for Parkinson's Disease – ROMP.

Participants answered all questions. In the process of linguistic and cultural translation and adaptation, there was no elimination of any question, but conceptual adjustments were made, modifying apple puree for creamy food and adding the terms comprehensible with intelligible and incomprehensible with unintelligible.

Test-retest scores, mean, and standard deviation

The scores, means and the standard deviation for speech, swallowing and saliva are described in Table 2.

Table 2
Scores, mean and standard deviation of the test and retest.

Variables		Mean (SD)
Speech
	Test score	15.8 (± 7.0)
	Retest score	15. 9 (± 7.0)
Swallowing
	Test score	14. 6 (± 6.7)
	Retest score	14.5 (± 6.6)
Saliva
	Test score	13.6 (± 6.1)
	Retest score	14.3 (± 6.7)
Total
	Test score	43.2 (± 16.6)
	Retest score	43.3 (± 16.7)

• SD: standard deviation.

Reliability

Internal consistency of the three domains and total - Cronbach's α

The value of Cronbach's α for the total scores and the three subdomains of the instrument were above 0.96, indicating that the ROMP had excellent internal consistency, as shown Table 3.

Table 3
Cronbach's (α).

Variable	Cronbach's (α)
Speech	0.99
Swallowing	0.99
Saliva	0.96
Total	0.99

Reproducibility of the three domains and total - ICC

Table 4 shows the ICC values. The ICC of the total score and of the three subdomains ranged from 0.93 to 0.99 indicating a high level of agreement in the test-retest.

Table 4
Intraclass Correlation Coefficient.

Variable	Measures	Intraclass correlationa	95%CI		F Test with True Value 0
			Lower bound	Upper bound	Significance
Speech	Single	0.99b	0.99	0.99	0.00
	Average	0.99c	0.99	1.00	0.00
Swallowing	Single	0.98b	0.97	0.99	0.00
	Average	0.99c	0.98	0.99	0.00
Saliva	Single	0.93b	0.85	0.96	0.00
	Average	0.96c	0.92	0.98	0.00
Total	Single	0.99b	0.99	0.99	0.00
	Average	0.99c	0.99	0.99	0.00

• Two-way mixed effects model where people effects are random and measures effects are fixed.
• ^a
  Type C intraclass correlation coefficients using a consistency definition – the between-measure variance is excluded from the denominator variance;
• ^b
  The estimator is the same, whether the interaction effect is present or not;
• ^c
  This estimate is computed assuming the interaction effect is absent, because it is not estimable otherwise. CI: confidence interval; Significance: (p ≤ 0.01).

Spearman's correlation between the ROMP and Likert-type scale

Spearman's correlation between the three domains of the ROMP and the Likert-type scale was significant, according to the following results: the correlation coefficient of the subscale of speech was 0.73 (p = 0.00), the correlation coefficient of the subscale of swallowing was 0.59 (p = 0.01), and the correlation coefficient of the subscale of saliva control was 0.67 (p = 0.00).

Spearman's correlation between the ROMP and UPDRS II

Spearman's correlation between the three domains of the ROMP and the corresponding subscales of the UPDRS II was also statistically significant, with the following results: the correlation coefficient of the subscale of speech was 0.52 (p = 0.00), the correlation coefficient of the subscale of salivation was 0.81 (p = 0.00), and the correlation coefficient of the subscale of swallowing was 0.84 (p = 0.00).

Spearman's correlation between the ROMP and UPDRS III

Spearman's correlation between the ROMP speech self-evaluation domain and the subscale of speech of the UPDRS III was 0.64 (p = 0.00), and therefore, substantial.

DISCUSSION

The ROMP questionnaire is a simple, quick-to-apply and easy-to-understand instruction, and the scales of measurements have remained the same as in the studies by Kalf et al.¹² and Paulinelli et al.³¹. Its translation and adaptation to Portuguese proved it to be a valid questionnaire in its content and reliability in quantifying difficulty in speaking, swallowing and controlling of saliva in patients with PD. In the evaluation of linguistic equivalence, all the questions of the original version were retained in the Brazilian version, as none of the questions proved to be invalid. Only conceptual reformulations were carried out, modifying some terms aimed at a better cultural adaptation.

The internal consistency and test-retest reproducibility of the ROMP were much higher than 0.70, indicating a reliable questionnaire. This result was similar to the Parkinson's disease questionnaire-39 item version³², and other questionnaires already validated in Brazil; those using the self-evaluation of voice, such as the Quality of Life in Voice, the Participation Profile and Vocal Activities and the Vocal Disadvantage Index^12,31.

Besides it being very important to measure the aspects of the impact of speech, swallowing and saliva changes on the patient, to make the necessary referrals and to define the basis of the treatment, it may be valuable to compare these results with other scales²⁷. In the case of this questionnaire, there was a significant difference in the correlation between all the questions between the ROMP and the Likert-type scale, the ROMP and the subscales of speech, salivation and swallowing in the UPDRS II and the subscale of speech in the UPDRS III. The ROMP was able to detect the clinical alterations between these studied scales showing how much the protocol can differentiate the studied groups, assuring the reliability of the instrument. We observed that within a contextualized analysis of each patient, the ROMP questions were clinically useful and allowed a reliable evaluation on an individual level.

When analyzing the results of this questionnaire in relation to the others validated in Brazil, the usefulness, validity and practicality of using questionnaires to assess the impact of diseases on individuals’ perceptions can be seen again, especially when these questionnaires are specific to some change, such as, in this case, speech, swallowing and saliva changes. In addition to all these advantages, self-assessment shows us the point of view of the individual being treated, and not only the clinician's view^31,32.

Regarding the limitations of this study, since the patients evaluated were classified with H&Y 2 and H&Y 3, we hypothesize that, if patients with H&Y 4 and 5 were evaluated, they might have presented with a higher degree of impairment, as scored by the questionnaire. Therefore, to detect these possible differences, research in patients with more severe disease involvement would be useful. Also, the results of this study justify the continuity of this research that has been expanded for validation of the ROMP construct in patients with Parkinson's disease.

The importance of this study is focused on the fact that there are few self-assessment questionnaires that evaluate speech, swallowing and saliva control in patients with PD. The patients’ self-perception of these changes contribute to the accuracy of speech-language assessment, with implications for clinical practice and future research, which may contribute to a more accurate diagnosis and a more adequate therapeutic plan, improving communicative effectiveness, swallowing and control of saliva for a longer time and, consequently, improving the quality of life of these patients.

In conclusion, the Brazilian Portuguese version of the ROMP questionnaire is reliable and valid and can be used in speech and language practice on an individual level. Its application in patients with PD facilitates the identification of speech disorders, swallowing disorders and drooling in these patients, which is important for adequate referral, evaluation and timely rehabilitation.

References

• ¹
  Pals P, Van Everbroeck B, Grubben B, Viaene MK, Dom R, van der Linden C et al. Case-control study of environmental risk factors for Parkinson's disease in Belgium. Eur J Epidemiol. 2003;18(12):1133-42. https://doi.org/10.1023/B:EJEP.0000006639.05690.92
  » https://doi.org/10.1023/B:EJEP.0000006639.05690.92
• ²
  Palermo S, Bastos ICC, Mendes MFX, Tavares EF, Santos DCL, Ribeiro AFC. [Phonoaudiological assessment and intervention in Parkinson's disease: Clinical-epidemiological analysis of 32 patients]. Rev Bras Neurologia. 2009;out/dez;45(4):17-24. Portuguese.
• ³
  Ciucci MR, Ma ST, Fox C, Kane JR, Ramig LO, Schallert T. Qualitative changes in ultrasonic vocalization in rats after unilateral dopamine depletion or haloperidol: a preliminary study. Behav Brain Res. 2007 Sep;182(2):284-9. https://doi.org/10.1016/j.bbr.2007.02.020
  » https://doi.org/10.1016/j.bbr.2007.02.020
• ⁴
  Wildgruber D, Riecker A, Hertrich I, Erb M, Grodd W, Ethofer T et al. Identification of emotional intonation evaluated by fMRI. Neuroimage. 2005 Feb;24(4):1233-41. https://doi.org/10.1016/j.neuroimage.2004.10.034
  » https://doi.org/10.1016/j.neuroimage.2004.10.034
• ⁵
  Pennington S, Snell K, Lee M, Walker R. The cause of death in idiopathic Parkinson's disease. Parkinsonism Relat Disord. 2010 Aug;16(7):434-7. https://doi.org/10.1016/j.parkreldis.2010.04.010
  » https://doi.org/10.1016/j.parkreldis.2010.04.010
• ⁶
  D'Amelio M, Ragonese P, Morgante L, Reggio A, Callari G, Salemi G et al. Long-term survival of Parkinson's disease: a population-based study. J Neurol. 2006 Jan;253(1):33-7. https://doi.org/10.1007/s00415-005-0916-7
  » https://doi.org/10.1007/s00415-005-0916-7
• ⁷
  Felix VN, Corrêa SM, Soares RJ. A therapeutic maneuver for oropharyngeal dysphagia in patients with Parkinson's disease. Clinics (Sao Paulo). 2008 Oct;63(5):661-6. https://doi.org/10.1590/S1807-59322008000500015
  » https://doi.org/10.1590/S1807-59322008000500015
• ⁸
  Wintzen AR, Badrising UA, Roos RA, Vielvoye J, Liauw L, Pauwels EK. Dysphagia in ambulant patients with Parkinson's disease: common, not dangerous. Can J Neurol Sci. 1994 Feb;21(1):53-6. https://doi.org/10.1017/S0317167100048770
  » https://doi.org/10.1017/S0317167100048770
• ⁹
  Nagaya M, Kachi T, Yamada T, Igata A. Videofluorographic study of swallowing in Parkinson's disease. Dysphagia. 1998;13(2):95-100. https://doi.org/10.1007/PL00009562P
  » https://doi.org/10.1007/PL00009562P
• ¹⁰
  Kalf JG, de Swart BJ, Bloem BR, Munneke M. Prevalence of oropharyngeal dysphagia in Parkinson's disease: a meta-analysis. Parkinsonism Relat Disord. 2012 May;18(4):311-5. https://doi.org/10.1016/j.parkreldis.2011.11.006
  » https://doi.org/10.1016/j.parkreldis.2011.11.006
• ¹¹
  Postuma RB, Berg D, Stern M, Poewe W, Olanow CW, Oertel W et al. MDS clinical diagnostic criteria for Parkinson's disease. Mov Disord. 2015 Oct;30(12):1591-601. https://doi.org/10.1002/mds.26424
  » https://doi.org/10.1002/mds.26424
• ¹²
  Kalf JG, Borm GF, Swart BJM, Bloem BR, Zwarts MJ, Munneke M. Reproducibility and validity of patient-rated assessment of speech, swallowing, and saliva control in Parkinson's disease. Arch Phys Med Rehabil. 2011 Jul; (92):1152-58. https://doi.org/10.1016/j.apmr.2011.02.011
  » https://doi.org/10.1016/j.apmr.2011.02.011
• ¹³
  Evatt ML, Chaudhuri KR, Chou KL, Cubo E, Hinson V, Kompoliti K et al. Dysautonomia rating scales in Parkinson's disease: sialorrhea, dysphagia, and constipation: critique and recommendations by movement disorders task force on rating scales for Parkinson's disease. Mov Disord. 2009 Apr;24(5):635-46. https://doi.org/10.1002/mds.22260
  » https://doi.org/10.1002/mds.22260
• ¹⁴
  World Health Organisation. International classification of functioning, disability and health (ICF). Geneva: World Health Organization; 2000.
• ¹⁵
  Miller N, Noble E, Jones D, Burn D. Life with communication changes in Parkinson's disease. Age Ageing. 2006 May;35(3):235-9. https://doi.org/10.1093/ageing/afj053
  » https://doi.org/10.1093/ageing/afj053
• ¹⁶
  Miller N, Noble E, Jones D, Burn D. Hard to swallow: dysphagia in Parkinson's disease. Age Ageing. 2006 Nov;35(6):614-8. https://doi.org/10.1093/ageing/afl105
  » https://doi.org/10.1093/ageing/afl105
• ¹⁷
  Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000 Dec;25(24):3186-91. https://doi.org/10.1097/00007632-200012150-00014
  » https://doi.org/10.1097/00007632-200012150-00014
• ¹⁸
  Hughes AJ, Daniel SE, Kilford L, Lees AJ. Accuracy of clinical diagnosis of idiopathic Parkinson's disease: a clinico-pathological study of 100 cases. J Neurol Neurosurg Psychiatry. 1992 Mar;55(3):181-4. https://doi.org/10.1136/jnnp.55.3.181
  » https://doi.org/10.1136/jnnp.55.3.181
• ¹⁹
  Hoehn MM, Yahr MD. Parkinsonism: onset, progression and mortality. Neurology. 1967 May;17(5):427-42. https://doi.org/10.1212/WNL.17.5.427
  » https://doi.org/10.1212/WNL.17.5.427
• ²⁰
  Hair F Junior, Anderson RE, Tatham RL, Black WC. Análise multivariada de dados. Porto Alegre: Bookman; 2005.
• ²¹
  Pereira MG. Epidemiologia: teoria e prática. Rio de Janeiro: Guanabara Koogan; 1995.
• ²²
  Gordis L. Determinação da validade e confiabilidade de testes de diagnóstico e rastreamento. 4a ed. Rio de Janeiro: Revinter: 2010.
• ²³
  McHorney CA, Robbins J. The SWAL-QOL and SWAL-CARE outcomes tools for dysphagia. Rockville: ASHA; 2004.
• ²⁴
  Hogikyan ND, Sethuraman G. Validation of an instrument to measure voice-related quality of life (V-RQOL). J Voice. 1999 Dec;13(4):557-69. https://doi.org/10.1016/S0892-1997(99)80010-1
  » https://doi.org/10.1016/S0892-1997(99)80010-1
• ²⁵
  Gasparini G, Behlau M. Quality of life: validation of the Brazilian version of the voice-related quality of life (V-RQOL) measure. J Voice. 2009 Jan;23(1):76-81. https://doi.org/10.1016/j.jvoice.2007.04.005
  » https://doi.org/10.1016/j.jvoice.2007.04.005
• ²⁶
  Jacobson HB, Johnson A, Grywalski C, Sillbergleit A, Jacobson G, Benninger MS et al. The voice Handicap Index (VHI): development and validation. Am J Speech Lang Pathol. 1997;6(3):66-70. https://doi.org/10.1044/1058-0360.0603.66
  » https://doi.org/10.1044/1058-0360.0603.66
• ²⁷
  Behlau M, Alves Dos Santos LM, Oliveira G. Cross-cultural adaptation and validation of the voice handicap index into Brazilian Portuguese. J Voice. 2011 May;25(3):354-9. https://doi.org/10.1016/j.jvoice.2009.09.007
  » https://doi.org/10.1016/j.jvoice.2009.09.007
• ²⁸
  Fahn S, Elton R. Members of the updrs Development Committee. In: Fahn S, Marsden CD, Calne DB, Goldstein M, editors. Recent developments in Parkinson's disease. Florham Park: Macmillan Health Care Information; 1987. Vol. 2; p. 153-63.
• ²⁹
  Robinson JP, Shaver PR, Wrightsman LS. Criteria for scale selection and evaluation. In: Robinson JP, Shaver PR, Wrightsman LS, editors. Measures of personality and social psychological attitudes. San Diego: Academic Press; 1991.
• ³⁰
  Aaronson N, Alonso J, Burnam A, Lohr KN, Patrick DL, Perrin E et al. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res. 2002 May;11(3):193-205. https://doi.org/10.1023/A:1015291021312
  » https://doi.org/10.1023/A:1015291021312
• ³¹
  Paulinelli BR, Gama AC, Behlau M. Validation of the Vocal Performance Questionnaire in Brazil. Rev Soc Bras Fonoaudiol. 2012;17(1):85-91. https://doi.org/10.1590/S1516-80342012000100016
  » https://doi.org/10.1590/S1516-80342012000100016
• ³²
  Marinus J, Ramaker C, van Hilten JJ, Stiggelbout AM. Health related quality of life in Parkinson's disease: a systematic review of disease specific instruments. J Neurol Neurosurg Psychiatry. 2002 Feb;72(2):241-8. https://doi.org/10.1136/jnnp.72.2.241
  » https://doi.org/10.1136/jnnp.72.2.241

Publication Dates

• Publication in this collection
  May 2018

History

• Received
  02 Oct 2017
• Accepted
  08 Feb 2018