The Brazilian version of the telehealth usability questionnaire (telehealth usability questionnaire Brazil): translation, cross-cultural adaptation, and psychometric properties

SUMMARY

OBJECTIVE:

The objectives of this study were to translate and cross-culturally adapt the telehealth usability questionnaire into Brazilian Portuguese and to evaluate its psychometric properties.

METHODS:

This was a methodological validation study carried out in two phases. In phase 1, the telehealth usability questionnaire was cross-culturally adapted with 10 participants comprising the expert committee members, including 5 healthcare professionals with theoretical and practical knowledge of telehealth, 1 methodologist, and 4 translators. This phase was performed at Universidade Federal de Juiz de Fora Physiotherapy Clinic School. In phase 2, the psychometric properties of telehealth usability questionnaire Brazil were analyzed. This phase included in-person assessments at Márcio Cunha Hospital, Minas Gerais. The recruitment period for both phases was from April 2020 to February 2021. Content validity, reliability, internal consistency, and criterion validity were analyzed. The criterion validity was evaluated using correlation with a validated instrument: the system usability scale.

RESULTS:

The telehealth usability questionnaire was adequately translated and cross-culturally adapted. The telehealth usability questionnaire Brazil presented an excellent content validity index of 0.96 with percentages of understanding higher than 90%. The telehealth usability questionnaire Brazil demonstrated great internal consistency (α=0.94 and ω=0.94), excellent intra-rater reliability (intraclass correlation coefficient=0.85, 95%CI 0.75–0.91), no difference between the test and retest [T (0.425), p>0.673], and no proportional bias (p=0.205). There was a moderate correlation between telehealth usability questionnaire Brazil and the system usability scale (r=0.52, p<0.0001).

CONCLUSION:

The telehealth usability questionnaire was adequately translated and cross-culturally adapted into Brazilian Portuguese and showed adequate psychometric properties for use in telehealth clinical practice and research in Brazilian-Portuguese-speaking individuals.

KEYWORDS:
Validation study; Surveys and questionnaires; Telemedicine

INTRODUCTION

Telehealth provides healthcare using telecommunication and information technologies, including mobile phones, smartphones, and other communication devices¹. The use of telehealth to provide healthcare remotely has become increasingly frequent with the development of technologies¹,². Telehealth emerges as an alternative and complementary strategy for managing patients when it is difficult to reach the traditional health services infrastructure or when face-to-face contact is not feasible. It may include diverse teleconsultation systems, telediagnosis, telemonitoring, tele-education, and telerehabilitation². An exponential increase in telehealth use started in 2020 when systems were expanded globally in response to the coronavirus disease 2019 (COVID-19) pandemic².

Telehealth systems must be helpful for both users and healthcare professionals. Good usability has several benefits, including fewer frequent errors, reduced user training time, acceptance, and greater efficiency and productivity when operating a particular system³. Although the access, acceptability, quality, and cost of telehealth systems have been evaluated across varying devices, the assessment of systems usability still needs to be improved. However, no telehealth-specific instrument is available for the Brazilian Portuguese-speaking population⁴ despite the growth of telehealth systems use in the country⁴.

The telehealth usability questionnaire (TUQ) was developed in 2016 and combined items from existing telehealth-specific and general computer usability questionnaires⁵. The TUQ assesses the healthcare professional and patient's usability experience and covers factors influencing usability, including usefulness, interface friendliness, effectiveness, reliability, and satisfaction with varying telehealth devices⁵. The TUQ is freely available and the most used tool for telehealth usability assessment⁶. Due to the broad TUQ applicability in clinical practice and research on telehealth, the objectives of this study were (1) to translate and cross-culturally adapt the TUQ into Brazilian Portuguese and (2) to evaluate its psychometric properties regarding content and criterion validity, test-retest reliability, and internal consistency.

METHODS

Study design and participants

This was a cross-sectional, methodological study approved by the Human Research Ethics Committees of the Universidade Federal de Juiz de Fora and Hospital Márcio Cunha, Minas Gerais (approvals: 28613719.8.0000.5147 and 28613719.8.3002.8147, in April 3, 2020, and January 8, 2021). All participants signed an informed consent form before participating in the study. This study had two phases. First, the cross-cultural adaptation of the TUQ was carried out following the recommended methodology⁷. Second, analyses of the TUQ Brazil psychometric properties followed the recommendations of the Consensus-based Standards for selecting health Measurement Instruments (COSMIN)⁸. A convenience sample of 10 participants was invited to comprise the expert committee members: 5 healthcare professionals with theoretical and practical knowledge of telehealth, 1 methodologist⁹, and 4 translators. The expert committee members’ recruitment period was from April 10, 2020, to July 16, 2020.

In evaluating the TUQ Brazil psychometric properties, a convenience sample of individuals with recent telehealth experience within the last 6 months before participation in the study, either as a health professional or a patient, was included. Participants with difficulty with the Brazilian-Portuguese language were excluded for reasons of understanding the instruments used in the study. Participants were recruited in this phase from January 10, 2021, to February 21, 2021.

Instruments

Telehealth usability questionnaire

The TUQ assesses the utility and usability of the technology⁵. TUQ is easy to apply and contains 21 statements ranked on a one-to-seven-level Likert-type scale from “strongly disagree” to “strongly agree” or “not applicable.” The TUQ score is determined by the average score of (1–7) responses, excluding the non-applicable items. The higher the response average, the greater the usability of the telehealth system. The questionnaire is freely available on the Internet (https://ux.hari.pitt.edu/v2/portal/#/about).

System usability scale

The system usability scale (SUS) was applied to test the TUQ Brazil concurrent criterion validity. The SUS is a widely used and valid self-reported instrument to assess the usability of varying technology interfaces. The SUS consists of 10 statements scored on a five-level Likert-type scale with anchors for “strongly agree” and “strongly disagree.” Using a simple formula, its final score ranges from 0 to 100. Higher scores indicate better system usability¹⁰.

Cross-cultural adaptation of the telehealth usability questionnaire

The author of the TUQ original version was contacted, and permission to cross-culturally adapt to the Brazilian-Portuguese language was sought. The translation and cross-cultural adaptation were performed in five steps⁷. (i) Translation of the instrument by two bilingual independent translators who were not aware of the objectives of the study. One translator was a health professional to provide a clinical perspective in the translated version that represents the language used by the target population. The second translator was unfamiliar with the topic addressed, thus being better able to detect different meanings and possible ambiguities⁷. (ii) Two authors carried out synthesizing translated versions for consensus, evaluating semantic, idiomatic, conceptual, linguistic, and contextual discrepancies, and analyzing the structure, layout, instrument instructions, and the scope and adequacy of the expressions contained in the items. This phase aimed to reach a single combined version. In case of divergences, adaptations would be made until a consensus on the translation was reached. (iii) Reverse translation was conducted by two native English speakers translators without prior access to the TUQ. This step aimed to assess the extent to which the translated version reflects the content of the items, as proposed in the original version of the questionnaire. The authors and the reverse translators analyzed discrepancies between the two versions and compared the back-translated consensus version with the original instrument in English. (iv) An expert committee of 10 participants⁷,⁹,¹¹ conducted the content validity assessment. The expert committee comprised five health professionals and a methodologist previously contacted with theoretical and practical knowledge about telehealth, in addition to the four translators who participated in the previous steps. This step consolidated all questionnaire versions and developed a preliminary, pre-final version. The expert committee analyzed the semantic, idiomatic, conceptual, linguistic, and contextual discrepancies and the questionnaire structure, layout, and user instructions⁷,⁹,¹¹. The committee members also answered questions about the comprehension of items. The agreement was verified quantitatively using the content validity index (CVI)¹². The CVI was calculated using a four-point Likert-type scale, in which the sum of responses scored as three and four of each committee member are divided by the total number of responses. (v) The evaluation of the final version or pilot study was performed, when telehealth users were interviewed for the understanding of each TUQ Brazil item⁹,¹³. The participants were instructed to complete the questionnaire. They were interviewed to investigate their perception of each item and the answer they chose, being asked about their understanding of each statement in the instrument and the justification for the difficulty in understanding. Items that presented 10% or more of “non-comprehension” would be modified, based on the participants’ responses, to reach the highest understanding in the final version until a pre-established percentage of adjustment (comprehension) in all items was reached (≥90%)⁹,¹³.

Psychometric properties of the telehealth usability questionnaire Brazil

Data collection included online and face-to-face assessments to evaluate the psychometric properties during the cross-cultural adaptation process. The analysis of psychometric properties was conducted for internal consistency, and the test-retest method used repeated evaluation of the TUQ Brazil version 7–14 days after the first assessment for reliability. In addition, participants were asked to answer the SUS in the retest session.

Statistical analysis

Data were stored and analyzed using the SPSS^® version 22.0 and Jamovi^® 2.3.26. The content validity was tested using the CVI's quantitative agreement among expert committee members¹². A CVI ³0.80 was evidence of content validity¹². Test-retest reliability was investigated using the intraclass correlation coefficient (ICC)¹⁴ and paired t-test. ICC³0.80 were accepted as a standard of reliability¹⁵. Studies recommend a minimum of 50 participants as the sample size required to test reliability¹⁶. The level of agreement of the test-retest was also verified using Bland-Altman plotting¹⁴. Internal consistency was assessed by Cronbach's alpha and McDonald's omega. Cronbach's alpha index range is 0–1¹⁷, and values between 0.75 and 0.95 were considered appropriate¹⁷. The McDonald's omega ³0.7 was considered adequate internal consistency¹⁴. The concurrent criterion validity was tested using Spearman's correlation coefficient between the TUQ Brazil and the SUS. Correlation coefficients were interpreted according to Schober et al.¹⁸

RESULTS

The final version of TUQ Brazil kept a similar format to the original questionnaire and is available in the electronic repository: https://data.mendeley.com/datasets/p8d3xyvfnp/4. The questionnaire versions used for cross-cultural adaptation, the final version, and the summary with items of divergence are also available in the electronic repository. There was no denial by the invited experts, and the 10 invited members agreed to participate. The committee reported adequate operational equivalence with the original questionnaire format. The TUQ Brazil showed an excellent CVI of 0.96 with great agreement.

A convenience sample of 54 individuals with prior experience as telehealth systems users (i.e., 30 patients and 24 healthcare professionals) was included. Three individuals did not complete the study assessments due to loss of contact (dropout rate of 1.6%). Participants’ demographic characteristics and telehealth modalities experienced are shown in Table 1. The TUQ Brazil items’ comprehension percentage on the final version evaluation was above 98.7%. No item required modification following patients’ and healthcare professionals’ comprehension assessment, and no redesign was necessary. The TUQ Brazil demonstrated excellent internal consistency. The Cronbach's alpha after single-item removal ranged from 0.927 to 0.940, and no item influenced reliability when removed from the analysis. Furthermore, the McDonald's omega was regarded as adequate (0.941). The TUQ Brazil showed excellent intra-rater reliability (ICC=0.85, 95%CI 0.75–0.91). The paired t-test did not demonstrate significant differences between the test and retest [T (0.425), p>0.673]. The Bland-Altman plot showed agreement between test and retest assessments, and the data distribution was homoscedastic with no proportional bias (p=0.205) (Figure 1). The TUQ Brazil showed a moderate correlation (r=0.52, p<0.01) with the SUS (Figure 2).

Figure 1
The Bland-Altman plot on the agreement between the two telehealth usability questionnaire Brazil assessments. LOA: limit of agreement, upper (bias+1.96×standard deviation) and lower (bias-1.96×standard deviation).

Figure 2
Spearman correlation plot between the telehealth usability questionnaire Brazil and the system usability scale.

Table 1
Participants’ characteristics and telehealth modalities experienced (n=54).

Age (years)		33 (21–63)
Sex	Female	48 (88.9)
Education level	Complete elementary school	2 (3.7)
	Complete secondary school	14 (25.9)
	Complete higher education	38 (70.4)
Telehealth modality	Telemonitoring	17 (31.5)
	Teleconsultation	16 (29.6)
	Teleconsulting	9 (16.7)
	Tele-education	8 (14.8)
	Telerehabilitation	4 (7.4)
	Telediagnosis	0 (0.0)

• Data are presented as median (minimum to maximum) or absolute number and percentage, n (%).

DISCUSSION

This study translated and cross-culturally adapted the TUQ into Brazilian Portuguese and analyzed its psychometric properties. The final version of the TUQ Brazil is the first telehealth usability assessment tool validated in Brazilian Portuguese and presented adequate operational equivalence to the original questionnaire format. The TUQ Brazil is a versatile questionnaire that may help identify telehealth systems limitations, guide new strategies to overcome the challenges while implementing new systems, and create a suitable configuration to implement and use telehealth services in Brazil successfully. In addition, the TUQ has no segmentation or domains and examines fewer items compared with other TUQs⁶; this may facilitate its use in population studies and services in public health.

During the translation and cross-cultural adaptation process, expert committee members modified or replaced no items from the original version. Despite the few divergences in the translations, the final version was approved with an excellent agreement, above the recommended CVI level¹². The TUQ Brazil was easily understood according to the percentage of comprehension observed in the target population. This confirmed the questionnaire's adequacy in assessing the usability of telehealth systems among Portuguese-speaking Brazilians. Adequate understanding of the translated and cross-culturally adapted versions of the TUQ was also reported for its Spanish¹⁹ and Turkish²⁰ versions.

The internal consistency of the TUQ Brazil items demonstrated the homogeneity of the questionnaire. The TUQ Brazil Cronbach's alpha range was similar to that found in another cross-culturally adapted TUQ version²⁰ and compatible with the one reported by Parmanto et al.⁵ in the original version of the questionnaire (from 0.79 to 0.93). Of note, the TUQ Brazil internal consistency was close to the ones reported for other questionnaires on telehealth systems usability²¹–²³. A score greater than 0.70 is rated positive for an instrument registering group data¹⁵,¹⁷. The TUQ Brazil also showed excellent intra-rater reliability (ICC>0.80), similar to values reported for the TUQ Turkish version²⁰ and the mHealth APP usability questionnaire (MAUQ), which is a usability questionnaire for telehealth applications²³.

The TUQ Brazil showed a moderate correlation with the SUS. The non-parametric distribution of the TUQ Brazil overall score may explain the absence of a more robust correlation; however, the strength of the correlation between the TUQ Brazil and SUS was close to those reported between questionnaires used to assess the usability of e-health questionnaires and the SUS, including the MAUQ (r=0.64) and the Post-Study System Usability Questionnaire (PSSUQ) (r=0.67)²³. Conversely, a strong correlation was reported between the TUQ and the Telemedicine Satisfaction Questionnaire²⁴, as both are telehealth-specific questionnaires instead of a general technology usability scale like the SUS²⁰.

There are some study limitations to be addressed. A convenience sample of users with experience in varying telehealth systems modalities was included for external validity, which is an important practical feature of the TUQ. However, the lack of a uniform telehealth modality under assessment, such as teleconsultation, telediagnosis, or telemonitoring only, prevented the analysis of additional psychometric properties of the TUQ Brazil, including floor and ceiling effects, standard error of psychometric, and minimal detectable difference. Although a minimum sample of 50 participants recommended for reliability assessment was recruited¹⁶, the subjects participated in evaluating the final version and analyses of the reported psychometric properties, as no item required modification following patients’ and healthcare professionals’ comprehension assessment, and no redesign was necessary from the pre-final to the final version of the TUQ Brazil.

CONCLUSION

The TUQ Brazil is a translated and cross-culturally adapted tool for assessing the usability of telehealth systems in the Brazilian-Portuguese-speaking population and presents adequate psychometric properties, including criterion validity, test-retest reliability, and internal consistency. Further studies must explore additional psychometric properties of the TUQ Brazil, including responsiveness to telehealth system usability modifications.

• Funding: This work study was supported by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (grant number 001), Fundação de Amparo à Pesquisa de Minas Gerais, Programa de Pesquisa para o SUS (PPSUS) (grant number APQ-03921-17), and Conselho Nacional de Desenvolvimento Científico e Tecnológico, Brazil (grant number 424542/2018-8).

ACKNOWLEDGMENTS

The authors are grateful for the invaluable contribution of the participants.

REFERENCES

• ¹
  Waller M, Stotler C. Telemedicine: a primer. Curr Allergy Asthma Rep. 2018;18(10):54. https://doi.org/10.1007/s11882-018-0808-4
  » https://doi.org/10.1007/s11882-018-0808-4
• ²
  Hron JD, Parsons CR, Williams LA, Harper MB, Bourgeois FC. Rapid implementation of an inpatient telehealth program during the COVID-19 pandemic. Appl Clin Inform. 2020;11(3):452-9. https://doi.org/10.1055/s-0040-1713635
  » https://doi.org/10.1055/s-0040-1713635
• ³
  Queirós A, Silva A, Alvarelhão J, Rocha NP, Teixeira A. Usability, accessibility and ambient-assisted living: a systematic literature review. Univ Access Inf Soc. 2015;14:57-66. https://doi.org/10.1007/s10209-013-0328-x
  » https://doi.org/10.1007/s10209-013-0328-x
• ⁴
  Caetano R, Silva AB, Guedes ACCM, Paiva CCN, Ribeiro GDR, Santos DL, et al. Challenges and opportunities for telehealth during the COVID-19 pandemic: ideas on spaces and initiatives in the Brazilian context. Cad Saude Publica. 2020;36(5):e00088920. https://doi.org/10.1590/0102-311x00088920
  » https://doi.org/10.1590/0102-311x00088920
• ⁵
  Parmanto B, Lewis AN, Graham KM, Bertolet MH. Development of the telehealth usability questionnaire (TUQ). Int J Telerehabil. 2016;8(1):3-10. https://doi.org/10.5195/ijt.2016.6196
  » https://doi.org/10.5195/ijt.2016.6196
• ⁶
  Hajesmaeel-Gohari S, Bahaadinbeigy K. The most used questionnaires for evaluating telemedicine services. BMC Med Inform Decis Mak. 2021;21(1):36. https://doi.org/10.1186/s12911-021-01407-y
  » https://doi.org/10.1186/s12911-021-01407-y
• ⁷
  Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25(24):3186-91. https://doi.org/10.1097/00007632-200012150-00014
  » https://doi.org/10.1097/00007632-200012150-00014
• ⁸
  Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737-45. https://doi.org/10.1016/j.jclinepi.2010.02.006
  » https://doi.org/10.1016/j.jclinepi.2010.02.006
• ⁹
  Gjersing L, Caplehorn JR, Clausen T. Cross-cultural adaptation of research instruments: language, setting, time and statistical considerations. BMC Med Res Methodol. 2010;10:13. https://doi.org/10.1186/1471-2288-10-13
  » https://doi.org/10.1186/1471-2288-10-13
• ¹⁰
  Martins AI, Rosa AF, Queirós A, Silva A, Rocha NP. European Portuguese validation of the system usability scale (SUS). Proc Comp Sci. 2015;67(16):293-300. https://doi.org/10.1016/j.procs.2015.09.273
  » https://doi.org/10.1016/j.procs.2015.09.273
• ¹¹
  Epstein J, Santo RM, Guillemin F. A review of guidelines for cross-cultural adaptation of questionnaires could not bring out a consensus. J Clin Epidemiol. 2015;68(4):435-41. https://doi.org/10.1016/j.jclinepi.2014.11.021
  » https://doi.org/10.1016/j.jclinepi.2014.11.021
• ¹²
  Coluci MZ, Alexandre NM, Milani D. [Construction of measurement instruments in the area of health]. Cien Saude Colet. 2015;20(3):925-36. https://doi.org/10.1590/1413-81232015203.04332013
  » https://doi.org/10.1590/1413-81232015203.04332013
• ¹³
  Ortiz-Gutiérrez S, Cruz-Avelar A. Translation and cross-cultural adaptation of health assessment tools. Actas Dermosifiliogr (Engl Ed). 2018;109(3):202-6. https://doi.org/10.1016/j.ad.2017.09.012
  » https://doi.org/10.1016/j.ad.2017.09.012
• ¹⁴
  Hutz CS, Bandeira DR, Trentini CM. Psicometria. 1ᵃ ed. Porto Alegre: Artmed Editora; 2015.
• ¹⁵
  Reichenheim ME, Moraes CL. [Operationalizing the cross-cultural adaptation of epidemiological measurement instruments]. Rev Saude Publica. 2007;41(4):665-73. https://doi.org/10.1590/s0034-89102006005000035
  » https://doi.org/10.1590/s0034-89102006005000035
• ¹⁶
  Terwee CB, Bot SD, Boer MR, Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34-42. https://doi.org/10.1016/j.jclinepi.2006.03.012
  » https://doi.org/10.1016/j.jclinepi.2006.03.012
• ¹⁷
  Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307-10. PMID: 2868172
• ¹⁸
  Schober P, Boer C, Schwarte LA. Correlation coefficients: appropriate use and interpretation. Anesth Analg. 2018;126(5):1763-8. https://doi.org/10.1213/ANE.0000000000002864
  » https://doi.org/10.1213/ANE.0000000000002864
• ¹⁹
  Bibiloni N, Torre AC, Angles MV, Terrasa SA, Vazquez Peña FR, Sommer J, et al. [Validation of a Spanish questionnaire on telemedicine usability]. Medicina (B Aires). 2020;80(6):649-53. PMID: 33254109
• ²⁰
  Ozden F, Ozkeskin M, Sari Z, Ekmekçi O, Yuceyar N. The reliability and validity of the Turkish version of the telehealth usability questionnaire and the telemedicine satisfaction questionnaire in patients with multiple sclerosis. Neurol Sci Neurophysiol. 2021;38(3):173-9. https://doi.org/10.4103/nsn.nsn_41_21
  » https://doi.org/10.4103/nsn.nsn_41_21
• ²¹
  Bakken S, Grullon-Figueroa L, Izquierdo R, Lee NJ, Morin P, Palmas W, et al. Development, validation, and use of English and Spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc. 2006;13(6):660-7. https://doi.org/10.1197/jamia.M2146
  » https://doi.org/10.1197/jamia.M2146
• ²²
  Demiris G, Speedie S, Finkelstein S. A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemed Telecare. 2000;6(5):278-84. https://doi.org/10.1258/1357633001935914
  » https://doi.org/10.1258/1357633001935914
• ²³
  Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B. The mHealth app usability questionnaire (MAUQ): development and validation study. JMIR Mhealth Uhealth. 2019;7(4):e11500. https://doi.org/10.2196/11500
  » https://doi.org/10.2196/11500
• ²⁴
  Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the telemedicine satisfaction questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare. 2003;9(1):46-50. https://doi.org/10.1258/135763303321159693
  » https://doi.org/10.1258/135763303321159693

Publication Dates

• Publication in this collection
  13 Nov 2023
• Date of issue
  2023

History

• Received
  17 July 2023
• Accepted
  24 July 2023