Development, validation and international certification of a health portal for people with disabilities

Desarrollo, validación y certificación internacional de un portal de salud para personas con discapacidad

ABSTRACT

Objectives:

to develop, validate with specialists and internationally certify a virtual health portal, the D+Informação.

Methods:

a quantitative methodological study. Twenty-two expert judges validated the Portal in the health and computer science areas using the Content Validity Index and the Gwet test. International certification followed HONcode principles^® to ensure the trust of information.

Results:

judges of health and computer science validated the portal obtaining a minimum Content Validity Index in more than 75% of the topics, in addition to the evaluation of general agreement, 0.253; content evaluation, 0.193, and interface evaluation, 0.230. All presented the value of Gwet with p ≤ 0,001.

Conclusions:

the portal D+Informação was developed, validated, internationally certified and is hosted on this website: https://demaisinformacao.com.br

Descriptors:
Website; Access to information; Consumer Health Information; Disabled People; Digital inclusion

RESUMEN

Objetivos:

desarrollar, validar con especialistas y certificar internacionalmente un portal virtual de salud, el D+Informação.

Métodos:

estudio metodológico cuantitativo. La validación del Portal fue hecha por 22 jueces especialistas en las áreas de salud y de la informática utilizando el Índice de Validez de Contenido y el test de Gwet. La certificación internacional siguió los principios del HONcode^® para garantizar la confianza de las informaciones.

Resultados:

el Portal fue validado por los jueces de salud y de la informática obteniendo Índice de Validez de Contenido mínimo en más de 75% de los tópicos, además la evaluación de concordancia general, 0,253; evaluación de contenido, 0,193; y evaluación de la interface, 0,230. Todos presentaron el valor de Gwet con p ≤ 0,001.

Conclusiones:

el Portal D+Informação fue desarrollado, validado, certificado internacionalmente y está hospedado en la siguiente dirección electrónica: https://demaisinformacao.con.br

Descriptores:
Redes de Comunicación de Computadores; Acceso a la Información; Información de Salud al Consumidor; Personas con Discapacidad; Inclusión Digital

RESUMO

Objetivos:

desenvolver, validar com especialistas e certificar internacionalmente um portal virtual de saúde, o D+Informação.

Métodos:

estudo metodológico quantitativo. A validação do Portal foi feita por 22 juízes especialistas nas áreas da saúde e da informática utilizando o Índice de Validade de Conteúdo e o teste de Gwet. A certificação internacional seguiu os princípios do HONcode^® para garantir a confiança das informações.

Resultados:

o Portal foi validado pelos juízes da saúde e da informática obtendo Índice de Validade de Conteúdo mínimo em mais de 75% dos tópicos, além da avaliação de concordância geral, 0,253; avaliação de conteúdo, 0,193; e avaliação da interface, 0,230. Todos apresentaram o valor de Gwet com p ≤ 0,001.

Conclusões:

o Portal D+Informação foi desenvolvido, validado, certificado internacionalmente e encontra-se hospedado neste endereço eletrônico: https://demaisinformacao.com.br

Descritores:
Website; Acesso à Informação; Informação de Saúde ao Consumidor; Pessoas com Deficiência; Inclusão Digital

INTRODUCTION

Currently, digital social media are considered tools for promoting inclusion, allowing access to information and the exchange of experiences among people with disabilities, their families, and professionals(¹). Thus, these social media have established themselves as places of collective learning and interactivity, in which individuals can interact with each other, with the possibility of specialized professional guidance, without the need to leave their homes. Therefore, there is a demand for a population by reliable sources of information, which offer security and privacy of user data, without advertising purposes, especially for more vulnerable audiences such as people with disabilities(²-³). These resources may be particularly relevant in the context of the pandemic of COVID-19, as there are no restrictions on their use because they do not require physical contact, which facilitates access to information, research, and access to health services.

The current challenge is access to secure information based on scientific evidence, which favors the fight against fake news and promotes knowledge intending to expand skills for better management of health care and the adoption of healthier lifestyle habits(⁴-⁵).

To this end, a research group (Neurorehab) has developed virtual learning tools. Among them, there is a virtual portal of information on health and inclusion, which aims to contribute to the dissemination of scientific knowledge, promoting an environment of virtual learning and empowerment, as a way to support people with disabilities (PcD), family and practitioners.

OBJECTIVES

To develop, validate with experts and internationally certify a virtual health portal, the D+Informação.

METHODS

Ethical aspects

The Ethics and Research Committee approved this study, which followed the ethical standards of research involving human beings established by Resolution No. 466/12 of the National Council of Research Ethics of the Ministry of Health, which addresses ethics in research with human beings.

Design, period and place of study

It is a methodological study divided into three stages: development, validation by judges in health and computer science areas, and international certification of information. The study used the quality improvement framework - SQUIRE from the EQUATOR network. The development of the Portal lasted eight months, between July 2018 and March 2019, when it was made available for functionality testing before its official release.

Study protocol

Portal development

The study opted for the model proposed by a Brazilian researcher(⁶), who has also been cited in other studies to create the Portal(¹,⁷). The model recommends four stages for software development: scope definition, planning, production, and implementation(¹,⁶-⁷). It hired a company to develop the initial structure of the Portal, using WordPress (an open-source project used to create websites, blogs, or applications). Then, specialists of the areas produced the contents of the Portal, which were addressed and entered by the responsible research group. The posts followed a template, and the Portal team reviewed the Portal with a copyright term for publishing the information in the virtual environment.

Virtual Portal validation by specialists

After its development, expert judges from health and computer science areas received the portal D+Informação for validation. The study considered the criteria and score for classification of specialists judges according to Fehring to select the judges. Criteria are adapted and used by other researchers for validation of virtual platforms(¹,⁷-⁸). There was a request for complete higher education as a minimum requirement to be considered a judge in this study, preferably with experience in rehabilitation and participation in scientific events. These criteria establish classification by score, with a minimum of 5 points for participation in the survey.

The sample of judges was composed of 22 participants (thirteen from the health area and nine from the computer science area), which met the recommended minimum (five judges) for validation of virtual platforms(⁷-⁸). The self-application instruments used, made on the Survey Monkey platform^®, have questions related to the overall and contents evaluation by health professionals, and the overall and interface evaluation for computer science judges.

International certification of information

After the improvement of the Portal, according to the validation of the judges, the international certification process started. The certification of Health Information took place according to the flowchart of the HONcode Code of ethics^® of health on the Net Foundation-HON(⁹). The eight principles established by HONcode must be complied with: authority, complementarity, confidentiality, attribution, justifiability, transparency, financial disclosure, and advertising. In addition to these general principles, there is the principle of privacy because it is a collaborative platform.

Data analysis

For data analysis and validation of the Portal, the study used the Content Validity Index (CVI). There were 24 questions for the health judges related to overall and content printing, and 37 questions for the computer science judges regarding overall and interface evaluation, with a Likert scale from 1 to 5, in which: 1 - I totally agree; 2 - I agree; 3 - neither agree nor disagree; 4 - I disagree, and 5 - I totally disagree(¹⁰-¹¹). It determined a high level of agreement between the judges(¹²): CVI minimum of 0.80 (80% agreement). Researchers reviewed the criteria that obtained CVI less than 0.80. In addition to the statistical analysis through the CVI, the study considered the judges’ observations and suggestions inserted in the last open question of the instrument. It also used Gwet’s statistics first-order agreement coefficient (AC1)(¹³), which has the advantage of resistance concerning marginal homogeneity and prevalence trait(¹⁴). The study used the R programs (R Core Team, 2018, version 3.5.3) and Excel(¹⁵) for the concordance analysis. It adopted a significance level of 5%.

RESULTS

The study presents the results according to the three phases: development of the Portal, validation by specialist judges, and certification of the information.

1st phase: Portal Development

The virtual portal, D+Informação (Figure 1), was developed following the methodology described, using the three phases development model for software: definition of the scope, planning, production, and implementation.

Figure 1
Home page of the Portal D+Informação, Ribeirão Preto, São Paulo, Brazil, 2019

2nd Phase: Validation by specialist judges

Characterization of judges

To validate the content, 13 judges from the health area participated, meeting the minimum score required according to the inclusion criteria of the research. There was a predominance of females (69.23%; n = 9) among health judges, aged between 30 and 60 years, and a mean of 40.57 years (SD = 7.28). The graduation time varied between 8 and 35 years, with an average of 15.30 years (SD = 9.20) of graduation, and all worked with rehabilitation and claimed to have participated in some scientific event related to their area of activity in the last two years. As for the academic degree, the 13 participants had some postgraduate degree: 61.54% (n = 8), master’s degree; 30.77% (n = 4), PhD; and 7.69% (n = 1), postdoctoral.

In addition to the overall validation, nine judges from the computer area participated in the ergonomics and interface validation. All were male, aged between 28 and 57 years, and an average of 41.22 years (SD = 9.95). The average training time was 17.70 (SD = 10.96) years, and 100% worked with computer science or software development. Regarding the academic degree, 22.22% (n = 2) had specialization, 33.33% (n = 3), Master’s degree, and 22.22% (n = 2), doctorate. All stated participation in scientific events related to their area of activity in the last two years.

Validation of the overall impression

For the evaluation of the overall impression of D+Informação, all 22 judges (9 from Computer science and 13 from health) participated, who analyzed the criteria described in Table 1.

Table 1
Distribution of the criteria of the evaluation of the overall impression of D+Informação, according to the judges of health and computer science and according to the Content Validity Index, Ribeirão Preto, São Paulo, Brazil, 2019, N = 22

Criteria	CVI health judges	CVI computer science judges	Overall CVI	% agreement
1. Interface favors participation.	1.00	0.89	0.95	95.45
2. Presentation of content favors participation.	0.92	0.89	0.91	90.91
3. Figures and pictures help in understanding the theme of the Portal.	0.84	1.00	0.91	90.91
4. The Portal is indicated as an educational tool.	0.92	0.78	0.86	86.36
5. The limitations of the Portal do not exceed its usefulness.	1.00	0.89	0.95	95.45
6. The name of the Portal (D+Informação) is suitable.	0.77	0.67	0.68	68.18
7. I recommend the Portal.	1.00	1.00	1.00	100.00

Bold numbers refer to items that did not reach the minimum CVI; CVI - Content Validity Index.Source: D+Informação, 2019.

Concerning the overall assessment, the Portal obtained the general agreement of the judges (AC1 = 0.253, SE = 0.043, p ≤ 0.001, Gwet test). Only the topic related to the Portal name showed CVI less than 80% (68.18%).

At the end of the instrument, judges had the opportunity to give suggestions. Three suggested more clarity and illustrative figures for the PcDs, which were accepted and generated modifications on the Portal. Another judge recommended the insertion of government pages regulating the rights of PcDs. Thus, a tab on “laws and rights” was added. They suggested and carried out a spelling review of the texts of the Portal, as well as the insertion of an icon on child development in the Tab “A-Z disabilities.” Polls and comments boxes were added in each post to increase the interactivity suggested by three judges, where users can interact with each other and share content on their social networks. There was a suggestion to change the name of the Portal, but it was not accepted.

Validation of Portal Content

To validate the content of the D+Informação, the 13 health judges analyzed the criteria described in Table 2.

Table 2
Distribution of D+Informação content evaluation criteria, according to the health judges and according to the Content Validity Index, Ribeirão Preto, São Paulo, Brazil, 2019, N = 13

Criteria for content evaluation	CVI	% agreement
1. The goals of the Portal are clearly defined.	0.84	84.62
2. The Portal is consistent with the objectives to which it is proposed.	0.92	92.30
3. The content of the Portal is updated.	0.84	84.62
4. The content of the Portal presents logical organization.	0.84	84.62
5. The content is consistent with the target audience.	0.92	92.30
6. The information is clear and concise.	0.84	84.62
7. The texts are easy to read.	0.92	92.30
8. The presentation of figures and photos is relevant to the information included in the texts.	0.84	84.62
9. The grammar is used correctly.	0.92	92.30
10. The terms are used correctly.	1.00	100
11. The presentation of the content captivates the attention of users.	0.77	76.92
12. The Portal stimulates user participation.	0.69	69.23
13. The Portal allows learning through the exchange of experiences between users.	0.77	76.92
14. The Portal stimulates mutual support.	0.84	84.62

Bold numbers refer to items that did not reach the minimum CVI; CVI - Content Validity Index.

Regarding the content, the Portal obtained the general agreement of the health judges (AC1 = 0.1925; SE = 0.0288; p ≤ 0.0000 Gwet test). It obtained CVIs below 80% in three topics of the Instrument: 1) “the presentation of content captivates the attention of users” (76.92%); 2) “The Portal stimulates user participation” (69.23%) 3) “the Portal allows learning through the exchange of experiences between users” (76.92%).

One judge suggested a more attractive interface for children and adolescents, and another maintained the suggestion of less technical language for PcDs, as well as more illustrative figures. Such ideas have already been reviewed in the overall evaluation.

Ergonomic validation of the Portal interface

Nine computer judges validated the ergonomics of the D+Informação interface following the criteria described in Table 3.

Table 3
Distribution of the evaluation criteria of ergonomics and interface of D+Informação, according to the judges of Computer science and according to the Content Validity Index, Ribeirão Preto, São Paulo, Brazil, 2019, N = 9

Criteria	CVI	% agreement
1. Menu pages have titles and headers.	1.00	100
2. Menu panels have logical grouping criteria.	1.00	100
3. Menu option names are concise.	0.89	88.89
4. Menu structure decreases steps for selection.	0.89	88.89
5. Use of abbreviations is minimized in menus.	0.89	88.89
6. Icons are readable.	1.00	100
7. Icons are distinct and always have the same meaning.	1.00	100
8. Icons are economical from the point of view of space on the screens.	0.89	88.89
9. Grammar is used correctly.	0.89	88.89
9. User controls system navigation.	1.00	100
10. User can interrupt and resume a dialog.	0.89	88.89
11. User moves in the system quickly.	0.89	88.89
12. Error message phrases are concise and objective.	0.89	88.89
13. Data to be read is presented continuously.	1.00	100
14. It adopts meaningful and familiar codes to users.	0.67	66.67
15. Meanings of colors are respected in the codes.	0.78	77.78
16. Presentation of texts and style features is appropriate.	1.00	100
17. Visual codes associate different categories of data scattered across screens.	0.89	88.89
18. Items selected for change, update, or trigger are highlighted.	0.89	88.89
19. Any change in the situation of control objects is visually presented in a clear way to the user.	0.89	88.89
20. Controls and commands are visually different from the information presented on the screens.	0.89	88.89
21. Screens present only the data and information indispensable for the user in his task.	0.89	88.89
22. Informational density of windows/screens is reduced.	0.89	88.89
23. Design does not overload the memory.	0.89	88.89
24. Presentation space is diagrammed into small functional zones.	0.89	88.89
25. Arrangement of interaction objects in a dialog box follows logical order.	0.89	88.89
26. The links work properly.	1.00	100
27. System optimization is suitable for different bandwidths.	0.67	66.67
28. System works correctly in different browsers.	0.67	66.67

Bold numbers refer to items that did not reach the minimum CVI; CVI - Content Validity Index.

Regarding the interface, the Portal obtained the general agreement of the computer science judges (AC1 = 0.2297; SE = 0.0221; p ≤ 0.001, Gwet test). It obtained CVIs of less than 80% in four topics of the Instrument: 1) “The Portal adopts significant codes or familiar to users” (66.67%); 2) “Meanings of colors are respected in the codes” (77.78%); 3) “System optimization is suitable for different bandwidths” (66.67%) 4) “the system works correctly in different browsers” (66.67%).

Phase 3: certification of information

Certification of the portal by HONCode^® started in October 2019. The website was viewed and evaluated in compliance with the Hon principles, so the following modifications/changes were made as suggested by the Hon evaluators:

Principle 1 - Authority: the presentation of the main author and the qualifications of the editorial Manager of the Portal were included in the “Who We Are” tab;

Principle 3 - Confidentiality: Clarify that the website uses cookies, which has been indicated in the Privacy Policy;

Principle 4 - Attribution: the information was referenced and dated, and all D+Informação publications were reviewed. The review included the date of posting along with the date of the last modification of the page;

Principle 8 - Advertising: the Portal mentions that it does not have advertising links or banners. A text was inserted in the D+Informação policies communicating that the Portal does not have any form of advertising.

In addition, to contemplate the Health 2.0 principles, in the D+Informação terms of use, two items that advise the user about the privacy and authority of the Portal were included.

DISCUSSION

The D+Informação Portal was developed and implemented by a multidisciplinary team of professionals from nursing, computer science, journalism, psychology, and social communication areas, following the development steps described in the method of this study. To this end, interdisciplinarity is added to a source of information based on evidence and mutual sharing of experiences. Rehabilitation services and programs should, as early as possible, start working with interdisciplinary teams to assess people’s needs and capabilities, including adaptations and assistive technologies(¹⁵).

It was necessary to use the expertise of the judges to verify the adequacy and quality of the items to validate the Portal. To participate in the validation process, the judges had to present the minimum criteria required in this study, resulting in a score that would confirm the expertise and experience necessary to improve this virtual instrument. Other researchers also used the scoring system as a strategy to select the judges who would participate in validation studies, ensuring the level of knowledge and experience on the subject that would be necessary for the evaluation and improvement of the developed tools(¹⁰,¹⁶). In this line, the present study corroborates the need to establish criteria and scores for the selection of qualified judges according to the area and theme of the digital educational tool to be developed.

All computer science and health judges collaborated on the overall impression of the D+Informação because, at this stage, the intention was to submit the Portal for overall analysis, that is, not specific to a particular area. According to the judges’ evaluations, only the criterion related to the name of the portal D+Informação did not reach the minimum CVI (0,80). After a dispute with the research group that developed the Portal, they decided not to change the name since this derives from previous projects, which developed virtual tools such as the social network D+Eficiência (https://demaiseficiencia.com/) a support network for patients, families, and professionals. Thus, to maintain the current audience without losing its brand, the name of the Portal remained D+ Informação, but the search link used was https://demaisinformacao.com.br as suggested by judges number 3, 8 and 9.

The space for comments and suggestions that judges had at the end of each instrument proved essential: it was a way to extract an even more valuable collaboration from extremely experienced judges for the Portal improvement. Following the judges’ suggestions, changes happened in language, grammar, content, including increasing interactivity and user participation in the Portal. To that end, publications gained polls and comment boxes, and the insertion of podcasting and games is in the planning phase to bring current topics to debates within D+Informação.

The evaluation of the content carried out by the health judges showed approval for most topics. Only three content topics did not reach the minimum CVI, related to the exchange of experience, attractiveness, and participation. These results led to pertinent changes in the structure and content of the Portal, inclusion of diverse contents for people with disabilities and their families, such as the latest news tab, as well as more reports of experiences. The study added the poll “What would you like to see on our Portal”? to stimulate user participation, aiming to meet the desires of the target audience and maintenance of active users. Scholars widely use interactive platform validations to ensure the relationship between the platform and its user(¹⁷-¹⁸). The content validation by health experts on the subject of the study was fundamental to ensure the quality of the information and its suitability to the intended audience.

The ergonomic evaluation of the interface by the judges of the computer science area generated a series of functional changes, which favored the usability and accessibility of the Portal. In contrast, according to the evaluation of these experts, this educational tool presented itself within the parameters of validity concerning interface ergonomics, evaluated as accessible and easy to browse for users. In this evaluation, only four topics evaluated did not reach the minimum required CVI (0.80). Of these, two were related to codes and colors of the Portal: “The Portal adopts significant or familiar codes to users”; “The usual meanings of colors are respected in the defined color codes”. This result led to the portal codes review, making it easier for the user to browse the page; the color scheme was standardized based on the D+Infomação logo. These changes contributed to a participatory design process, recommended by international researchers as a way to avoid complications related to poor health research interface design(¹⁹).

The other two topics that did not reach the minimum CVI in the computer science judges’ assessment referred to the Internet portal Navigation System: “System Optimization is suitable for different bandwidths”; “the system works correctly in different browsers”. In these two topics, the low CVI was related to the difficulty of evaluating the judges in several different systems and bands, so some pointed to the option “neither agree, nor disagree”. Therefore, the Portal has been tested, verified, and made available in different bandwidths and all browsers and devices with internet access.

The Portal obtained significant agreement in the three evaluations carried out (overall evaluation, content evaluation, and ergonomic evaluation of the interface), using the AC1 statistics developed by Gwet. Although the CVI showed the observation of the relative frequency, the AC1 Gwet is more comprehensive since it considers the diverse categorization of the frequency distribution of the Likert scale. Thus, it is increasingly common to use this statistical test in studies that validate an instrument(²⁰-²¹). Researchers have used judge validation for the technology development process as an essential method for adapting the tools to the target audience(²²). This process allows evaluators to point out the weaknesses in the development of virtual portals and make suggestions based on technical experiences the characteristics of the target audience. To that end, the recommendations given by the judges in this study were pertinent to complement the quality of the information disclosed to users.

D+Informação certification by HONcode^® was productive because it promoted essential modifications for its suitability, raising its level following international standardization and certification. The process to achieve certification required considerable effort from the entire Portal development team to make the adjustments in line with HONcode’s eight principles(²³). Conversely, the results showed that judges requested few changes after the Portal evaluation by the experts of HON. It happened because the website was constructed according to the principles of HONcode from the very beginning. Therefore, such experience has shown that starting the development of digital tools following an international standardization can facilitate the acquisition of certifications in the future.

Certification of virtual tools with the use of HONcode^® has been widely adopted by other health scholars worldwide(²⁴-²⁸). More than 7 thousand health websites based in 102 countries use the HON code to be certified (HONcode, 2020). The HONcode seal^® certifies that the Portal has high-quality, clear, and reliable online information for users. The HONcode^® Certification is valid for one year. Thus, the portal D+Informação must go through the entire certification process again to maintain its international recognition annually. In the future, to increase excellence, the Portal may be submitted to other international certifications, such as American Accreditation Healthcare Commission (URAC-sites - Utilization Review Accreditation Commission) and the Agencia de Calidad Sanitaria de Andalucia (“sanitary Web pages”).

Study limitations

Regarding the study limitations, the study analyzed the eight main pages of the Portal to evaluate the validation and certification processes. Because it is a large portal, it did not assess the numerous subtopics composed mainly of posted materials. To complement this internalized evaluation of the Portal, a future study of the research group will evaluate it from the user’s perspective after the individual experience of surfing on all subtopics. Another limitation is that different browsers and bands failed to be tested effectively by the judges. Despite this, after the launch, no user reported navigation difficulties faced by the team during the tests carried out on different browsers and bands.

Contributions to the area of Nursing, Health or Public Policy

The rigorous development methods and validation process of the Portal allowed the success of the developed technological tool and reproduced a scientific character to the development process, differentiating it from other existing health websites. In conclusion, the improvement carried out during the validation of D+Informação collaborated for the dissemination of knowledge in health and inclusion, based on scientific and clinical expertise offering clear, attractive, and accessible information to the population.

Therefore, the expectation is that the D+Informação Portal will become a democratic, helpful and reliable source of information on health and inclusion for all Brazilians, especially for PcDs, within an interactive, participatory, and internationally certified environment.

CONCLUSIONS

The portal D+Informação has been developed, validated, internationally certified, and is hosted at the following website address: https://demaisinformacao.com.br/ The careful method for the development of the tool provides a scientific character to this whole process, differentiating it from other existing health sites.

SUPPLEMENTARY MATERIAL

Cintra, Michel Marcossi. Development, validation, accessibility analysis and international certification of a health and inclusion information portal [dissertation]. Ribeirão Preto: University of São Paulo, School of Nursing of Ribeirão Preto; 2020 [cited 2021-10-02]. https://doi.org/10.11606/D.22.2020.tde-18032021-091718

• FUNDING
  This study was funded by the program CAPES-DAAD - Probral Notice Nº14 / 2019 process Nº 23038.005586 / 2019-95.

REFERENCES

• ¹
  Favoretto N, Faleiros F, Lopes F, Freitas G, Käppler C. Online health forum as a support for people who perform intermittent vesical catheterization. Texto Contexto Enferm. 2019;28(1):e20180263. https://doi.org/10.1590/1980-265X-TCE-2018-0263
  » https://doi.org/10.1590/1980-265X-TCE-2018-0263
• ²
  Frossard VC, Dias MCM. O impacto da internet na interação entre pacientes: novos cenários em saúde. Interface (Botucatu). 2016;20(57):349-61. https://doi.org/10.1590/1807-57622014.1334
  » https://doi.org/10.1590/1807-57622014.1334
• ³
  Rus HM, Cameron LD. Health communication in social media: message features predicting user engagement on diabetes-related Facebook pages. Ann Behav Med. 2016;50(5):678-89. https://doi.org/10.1007/s12160-016-9793-9
  » https://doi.org/10.1007/s12160-016-9793-9
• ⁴
  Frias Filho O. O que é falso sobre fake news. Rev USP. 2018;(116):39-44. https://doi.org/10.11606/issn.2316-9036.v0i116p39-44
  » https://doi.org/10.11606/issn.2316-9036.v0i116p39-44
• ⁵
  Silva MAR, Melo SA. Fake news: fronteiras do jornalismo e circulação de (des)informação sobre saúde. Rev Eletron Comunic Inf Inov Saude. 2020:14(1):1-5. https://doi.org/10.29397/reciis.v14i1.2047
  » https://doi.org/10.29397/reciis.v14i1.2047
• ⁶
  Bernardo V. Metodologia para desenvolvimento de projeto multimídia aplicado ao ensino da medicina [Dissertação]. São Paulo: Universidade Federal de São Paulo; 1996.
• ⁷
  Góes FSN. Desenvolvimento e avaliação de objeto virtual de aprendizagem interativo sobre o raciocínio diagnóstico em enfermagem aplicado ao recém-nascido pré-termo [Tese]. Ribeirão Preto: Universidade de São Paulo; 2010.
• ⁸
  Cucick CD. Desenvolvimento e validação de vídeo educativo para autocateterismo vesical intermitente limpo. Rev Eletron Enferm. 2019;21:53973 https://doi.org/10.5216/ree.v21.53973
  » https://doi.org/10.5216/ree.v21.53973
• ⁹
  Alexandre NMC, Coluci MZO. Content validity in the development and adaptation processes of measurement instruments. Cien Saude Colet. 2011;16(7):3061-8. https://doi.org/10.1590/S1413-81232011000800006
  » https://doi.org/10.1590/S1413-81232011000800006
• ¹⁰
  Campos Rivera NH, Sotelo Quiñonez TI. Design and validation of a scale of maternal attitudes towards childhood overweight and obesity. Acta Colomb Psicol. 2019;22(2):148-62. https://doi.org/10.14718/acp.2019.22.2.8
  » https://doi.org/10.14718/acp.2019.22.2.8
• ¹¹
  Martins MC, Veras JEGLF, Uchoa JL, Pinheiro PNC, Vieira NFC, Ximenes LB. Food safety and the use of regional foods: the validation of a serial album. Rev Esc Enferm USP. 2012;46(6):1354-61. https://doi.org/10.1590/S0080-62342012000600011
  » https://doi.org/10.1590/S0080-62342012000600011
• ¹²
  Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008;61:29-48. https://doi.org/10.1348/000711006X126600
  » https://doi.org/10.1348/000711006X126600
• ¹³
  McCray G. Assessing inter-rater agreement for nominal judgement variables. Paper presented at: Language Testing Forum; 2013 Nov 15-17; Nottingham, England.
• ¹⁴
  R Core Team. A language and environment for statistical computing [Internet]. Vienna: R Foundation for Statistical Computing. 2018.Available from: http://www.R-project.org/
  » http://www.R-project.org/
• ¹⁵
  Health On the Net [Internet]. HON; 2019[cited 2020 Mar 22]. Available from: http://www.hon.ch/
  » http://www.hon.ch/
• ¹⁶
  Wild CF, Nietsche EA, Salbego C, Teixeira E, Favero NB. Validation of educational booklet: an educational technology in dengue prevention. Rev Bras Enferm. 2019;72(5):1385-92. https://doi.org/10.1590/0034-7167-2018-0771
  » https://doi.org/10.1590/0034-7167-2018-0771
• ¹⁷
  Bacallao-Pino LM. Recursos interactivos y redes sociales en la información sobre salud sexual y reproductiva en la prensa cubana: la sección “Sexo sentido", del diario Juventud Rebelde. Rev Cuba Inf Cienc Salud. 2015[cited 2020 Abr 15];26(2):94-106. Available from: http://scielo.sld.cu/scielo.php?script=sci_arttext&pid=S2307-21132015000200002
  » http://scielo.sld.cu/scielo.php?script=sci_arttext&pid=S2307-21132015000200002
• ¹⁸
  Santiago DCS, Coutinho BD, Silva ASR. Uso de TICs: experiência a partir da extensão universitária. Ext Acao. 2017;2(14):108-17. https://doi.org/10.32356/exta.v2.n14.20315
  » https://doi.org/10.32356/exta.v2.n14.20315
• ¹⁹
  Hyochol A, Michael W, Debra L, Eunyoung CRN, Roger BFP. Depressão e dor em asiáticos americanos e brancos com osteoartrite do joelho. Physiol Behav. 2017[cited 2020 May 15];176(10):139-48. https://doi.org/10.1016/j.physbeh.2017.03.040
  » https://doi.org/10.1016/j.physbeh.2017.03.040
• ²⁰
  Alves MG, Batista DFG, Cordeiro ALPC, Silva MD, Canova JCM, Dalri MCB. Production and validation of a video lesson on cardiopulmonary resuscitation. Rev Gaucha Enferm. 2019;40:e20190012. https://doi.org/10.1590/1983-1447.2019.20190012
  » https://doi.org/10.1590/1983-1447.2019.20190012
• ²¹
  Grillo ACS, Faleiros F, Silva JCF, Tate DG, Greve JMA, Tholl AD. Portuguese language translation of the lower urinary tract data set for patients with spinal cord injury Texto Contexto Enferm. 2018;27(4):e5390016. https://doi.org/10.1590/0104-07072018005390016
  » https://doi.org/10.1590/0104-07072018005390016
• ²²
  Rosa BVC, Girardon-Perlini NMO, Gamboa NSG, Nietsche EA, Beuter M, Dalmolin A. Development and validation of audiovisual educational technology for families and people with colostomy by cancer. Texto Contexto Enferm. 2019;28:e20180053. https://doi.org/10.1590/1980-265x-tce-2018-0053
  » https://doi.org/10.1590/1980-265x-tce-2018-0053
• ²³
  Abdulrahman SA, Ganasegeran K, Surajudeen AA, Kurubaran G. m-Health in public health practice: a constellation of current evidence. In: Jude HD, Balas VE, editors. Telemedicine technologies: big data, deep learning, robotics, mobile and remote applications for global healthcare. London: Elsevier; 2019. p. 171-82 https://doi.org/10.1016/B978-0-12-816948-3.00011-8
  » https://doi.org/10.1016/B978-0-12-816948-3.00011-8
• ²⁴
  Bragazzi NL, Prasso G, Re TS, Zerbetto, Del Puente G. A reliability and content analysis of Italian language anorexia nervosa-related websites. Risk Manag Healthc Policy. 2019;12:145-51. https://doi.org/10.2147/RMHP.S193088
  » https://doi.org/10.2147/RMHP.S193088
• ²⁵
  Huang Z, Lum E, Car J. Medication management apps for diabetes: systematic assessment of the transparency and reliability of health information dissemination. JMIR Mhealth Uhealth. 2020;8(2):e15364. https://doi.org/10.2196/15364
  » https://doi.org/10.2196/15364
• ²⁶
  Murray KE, Murray TE, O’Rourke AC, Low C, Veale DJ. Readability and quality of online information on osteoarthritis: an objective analysis with historic comparison. Interact J Med Res. 2019;8(3):e12855. https://doi.org/10.2196/12855
  » https://doi.org/10.2196/12855
• ²⁷
  Neunez M, Goldman M, Ghezzi P. Online information on probiotics: does it match scientific evidence?. Front Med. 2020;6:296. https://doi.org/10.3389/fmed.2019.00296
  » https://doi.org/10.3389/fmed.2019.00296
• ²⁸
  Passos KK. Quality of information about oral cancer in Brazilian Portuguese available on Google, Youtube, and Instagram. Med Oral Patol Oral Cir Bucal. 2020;25(3):e346-52. https://doi.org/10.4317/medoral.23374
  » https://doi.org/10.4317/medoral.23374

Edited by

EDITOR IN CHIEF: Antonio José de Almeida Filho

ASSOCIATE EDITOR: Priscilla Valladares Broca

Publication Dates

• Publication in this collection
  22 Apr 2022
• Date of issue
  2022

History

• Received
  01 Mar 2021
• Accepted
  22 Sept 2021