Assessment of the quality of information on Covid-19 websites: an alternative for combating fake news

ABSTRACT

Information and communication technologies have spread worldwide and are increasingly present in almost every aspect of human life, including health. The issue of information quality has assumed great importance amid the widespread dissemination of false knowledge, especially in a pandemic. We evaluated the quality of information on Covid-19 websites of four Health Secretariats in the state of Mato Grosso do Sul, Brazil, from July to August 2020. This participatory evaluation employed five criteria, following the international literature, namely: Technical, Interactivity, Comprehensiveness, Readability, and Accuracy, subdivided into 46 indicators. The results of the evaluated websites point to a low level of compliance with the indicators and criteria adopted and fail to disclose the primary scientific evidence on the topic, available on the Ministry of Health’s website. The high circulation of fake news has marked the Covid-19 pandemic. In this context, Health Secretariats’ pages should display quality and intelligible information about the disease. Only then will they offer reliable informational content based on scientific evidence, contributing to the fight against fake news and its adverse impacts.

KEYWORDS
Internet; Evaluation studies as topic; Coronavirus infections; Online access to information; Quality of health information

RESUMO

As tecnologias de informação e comunicação têm-se disseminado globalmente. Elas estão cada vez mais presentes em quase todos os aspectos da vida humana, incluindo a saúde. Em meio à ampla disseminação de informações falsas, a questão da qualidade da informação tem assumido grande importância, especialmente em contexto de pandemia. Entre julho e agosto de 2020, foi realizada uma avaliação da qualidade da informação em sites sobre Covid-19 de quatro secretarias de saúde do estado de Mato Grosso do Sul. Essa avaliação participativa utilizou cinco critérios, acompanhando a literatura internacional, a saber: Técnico, Interatividade, Abrangência, Legibilidade e Acurácia, subdivididos em 46 indicadores. Os resultados apontam que os sites avaliados apresentaram baixo grau de conformidade com os indicadores e critérios utilizados e não divulgam as principais evidências científicas sobre o tema, disponíveis no site do Ministério da Saúde. A pandemia de Covid-19 tem sido marcada pela alta circulação de fake news. Nesse contexto, é imprescindível que páginas de secretarias de saúde apresentem informação sobre a doença com qualidade e legibilidade. Só assim, oferecerão um conteúdo informativo confiável e baseado em evidências científicas, contribuindo para o enfrentamento de notícias falsas e seus impactos negativos.

PALAVRAS-CHAVE
Internet; Estudos de avaliação como assunto; Infecções por coronavirus; Acesso à informação on-line; Qualidade da informação em saúde

Introduction

Since the end of the 20^th century, Information and Communication Technologies (ICT) disseminated through the internet are increasingly blending into almost every aspect of human life. Their global popularization has contributed to changes in practices and actions in different fields¹. The internet provides countless sources of information. Furthermore, individuals can produce information by organizing their website, blog, page, or profile on social media platforms. Many people can interact with each other through different online tools and establish social networks, even if they are geographically distant².

Nowadays, different public and private organizations or individuals produce information on topics linked somehow to health-disease issues. These data are often insufficient, unsatisfactory, obsolete, incorrect, or incomprehensible. Thus, they can put the health of citizens and society at risk. In this context, assessing the quality of health information available online started to assume a relevant role globally³.

In the original term, the current global setting is significantly impacted by the Covid-19 pandemic – or Coronavirus Disease 2019. As of April 2021, more than 132 million confirmed Covid-19 cases were accounted for, with more than 2.8 million deaths from the disease worldwide⁴. More than 13 million cases were confirmed in Brazil as of the beginning of that same year, totaling approximately 337 thousand deaths⁵. The emergence and dissemination of Covid-19 produced a series of impacts in different spheres of society, directly or indirectly affecting millions of people around the planet⁶.

Countless challenges resulting from the pandemic have affected public health, such as adopting several preventive measures, such as the use of masks and alcohol gel and social distancing. The increase in cases sometimes led to the collapse of health systems, particularly in emergency services and hospital intensive care centers⁷. The health professionals providing care often suffer from anxiety and depression due to exhausting work⁸. We should stress that this pandemic mainly affects the socially vulnerable⁹.

Evidence-based online health information can, for example, reduce the use of health services¹⁰. According to Lagan, Sinclair, and Kernohan¹¹, they can have “a visible impact on women’s decision-making concerning all aspects of their pregnancy”^11(336). Thus, quality online information from a reliable, up-to-date, and easily understandable source can play a crucial role in self-care practices, service costs, disease prevention, and health promotion¹¹,¹². With this information at hand, in the case of Covid-19, citizens will know and possibly adopt the preventive measures recommended by science and where and when to perform diagnostic tests, and receive possible and necessary medical care¹³.

In the current context, digital media, characterized by a global, instantaneous, and capillary presence, have become crucial and singled out the current pandemic from those that existed until then¹³,¹⁴. Unlike other historical moments, individuals can have immediate and remote access to updated news about the evolution of the pandemic. In an online survey of the most accessed information sources about Covid-19, Ho, Chen, and Yen¹⁵ concluded that:

The most common sources of information related to Covid-19 were internet media (80.52%), traditional media (52.62%), relatives (24.36%), co-workers (23.57%), friends (21.08%), academic courses (20.18%), and medical staff (19.03%)¹⁵⁽²⁾.

While it has escalated, the current pandemic has not inaugurated the search for online health information. In recent decades, the number of individuals using ICT to learn more about their health condition has increased¹⁶. This practice has been driven and is becoming strategic among patients suspected or diagnosed with Covid-19 due to the context of uncertainty produced by the pandemic, characterized by “devastating losses of lives, physical morbidity, mental anguish, and general distress”¹⁷⁽²⁾.

False news about Covid-19 proliferates before the growing demand for information on the internet. These are fake news in different formats, which usually have an affirmative text. This characteristic leads many people, who typically do not check the information, to believe and share such content¹⁸.

Pereira Neto, Barbosa, and Flynn¹⁸ warn that cases of media misinformation are not new. On the contrary, they have been present since the first writing systems, mainly motivated by financial or ideological reasons, especially after a media concentration in a few large companies. However, ICT facilitated the creation and dissemination of imprecise and misleading news, which could partly explain the dimension of misinformation in recent years.

The pandemic has infinitely multiplied the number of websites, blogs, and YouTube channels that offer up-to-date information and present views of opinion makers aimed at increasingly specific audiences¹⁸⁽¹²⁾.

This trend has increased the possibility of producing fake news about the pandemic. Some studies were recently published evaluating the characteristics and consequences of fake news in the case of Covid-19. Among them, the study by Vasconcelos-Silva and Castiel¹⁹ stands out, in which the authors claim that fake news reveals itself as

persecutory, naive, prejudiced tertiary appropriations, aligned with political, criminal, or merely commercial agendas, to validate ideas and concepts based on mistaken – or misleading – scientific viewpoints¹⁹⁽²⁾.

Galhardi et al.²⁰ admit that

the dissemination of fake news contributes to the discredit of science and global public health institutions and weakens people’s adherence to the necessary preventive care when dealing with the epidemic^20(4208).

None of these papers devote their attention to the reaction and resistance to this trend or present or analyze the possible and available alternatives to address this issue. One of them can be translated as fact-checking²¹. These are information agencies dedicated to fact-checking. They are currently accredited by the International Fact-Checking Network (IFCN) under the same code of principles. The IFCN is a network that aggregates fact-checking initiatives worldwide. The National Network to Combat Disinformation (RNCD) was recently created in Brazil. It gathers, on a single platform, projects, initiatives, research groups, observatories, and agencies across the country that monitor, identify, denounce and combat fake news²¹.

An alternative reaction is associated with the practice of literacy-based education²². Sousa et al.²³ admit that digital literacy is a modality of informational competence that allows users to have less difficulty and more skills in searching for and distinguishing information and knowing how to apply them to improve their health and quality of life.

This paper presents and discusses another way to react to fake news. This is the assessment of the quality of information available on health websites³. It specifically describes and analyzes the result of the quality assessment of four public websites on Covid-19: one linked to the Mato Grosso do Sul State Health Secretariat and three municipal health secretariats from the same federation, namely, Campo Grande, Corumbá, and Dourados.

The research that served as the basis for the preparation of this paper was carried out by a group of professors and students during the online Special Topics Discipline: Evaluation of the Quality of Information on Infectious and Parasitic Diseases Websites, of the Infectious and Parasitic Diseases (PPGDIP) Postgraduate Program of the Federal University of Mato Grosso do Sul (UFMS), from July 8 to August 19, 2020.

The methodology, criteria, and indicators used and the results obtained in evaluating the quality of information on Covid-19 websites of Brazilian government institutions will be presented below.

Material and methods

Criteria built from two systematic reviews were employed to construct the criteria and indicators used to assess the quality of information on Covid-19 of Brazilian government websites. The first identified and analyzed 79 papers published up to 2001²⁴. The second was based on 279 papers published from 2001 to 2014, available in international bibliographic databases²⁵.

Both reviews concluded that most studies published in the world used five criteria to assess the quality of information on health websites: technical, interactivity, comprehensiveness, readability, and accuracy. The technical criteria aim to identify whether or not there is a person responsible for the site or the information offered has a source. It also checks, for example, the date of creation and update of this virtual environment. The interactivity criterion is available to verify whether the site provides means of communication between users and content producers, whether it has a ‘Contact Us’ channel, it offers access to social networks, and, for example, search tools and accessibility alternatives for people with disabilities²⁶ are available. The readability criterion assesses whether the information is arranged to ensure the understanding of lay users. The comprehensiveness criterion looks at whether the site offers the set of information relevant to a particular topic or health problem. If it is the case of evaluating the website’s information on communicable disease, the sense of comprehensiveness includes information on prevention, symptoms, diagnosis, and treatment. The accuracy criterion measures the degree of agreement between the information offered on the website and the best evidence available or accepted by science. The study on the assessment of information quality on health websites presented in this work adopted the five criteria mentioned above.

Each criterion consisted of several indicators: specific evaluative questions, mostly with dichotomous structured answers (Yes or No). In the case of technical criteria, a question asked, for example, whether ‘it was updated in the last week’. Regarding comprehensiveness, a question asked whether ‘there is information about symptoms’. Regarding interactivity, a question asked whether the site ‘has any social network’. Concerning readability, one of the questions aimed to know whether the evaluator ‘had difficulty understanding the information on prevention’. We used information about Covid-19 of the official Ministry of Health (MS) website as a reference (https://coronavirus.saude.gov.br/) to build the indicators of the accuracy criterion. Only the accuracy criterion indicators allowed the ‘Yes’, ‘No’, or ‘Partially’ options as correct answers. They were not formulated as a question but as an affirmative sentence. Evaluators should verify whether the information mentioned in the indicator was fully or partially present or absent in the assessed website. One of the indicators used was: ‘A preventive measure is washing hands frequently with soap and water or sanitizing with 70% alcohol gel’. Then, the following question was included: ‘Is this information available on this website?’. The answers could be: ‘Yes’, ‘No’ or ‘Partially’. The technical and readability criteria consisted of six indicators each; comprehensiveness and interactivity for seven indicators each; and accuracy, 20 indicators. The five criteria adopted totaled 46 indicators. The positive response to each of these indicators reveals the level of compliance between the offered and ideal information. These 46 indicators make up the ‘gold standard’; they are part of the information set that an excellent site should have.

Box 1 presents the underlying indicators of each criterion. A large number of accuracy indicators is noteworthy (20), which is justified by the inclusion of indicators related to prevention (4), symptoms (5), transmission (4), diagnosis (3), and treatment (4) in this criterion.

Box 1.
Criteria and indicators used to assess the quality of information on Covid-19 websites

Criterion	Nº of indicators	Indicators
Technical (T)	6	Was the site updated in the last week? Has the source of information for prevention been presented? Has the source of information for the symptom been presented? Has the source of information for the transmission been presented? Has the source of information for the diagnosis been presented? Has the source of information for the treatment been presented?
Interactivity (I)	7	Is there a customer service channel for contact on the website? Does the website have any social networks? Is there a search engine on the website? Is the website page attractive? Is the website easy to navigate? Does the website have accessibility tools (Pounds, Autocontrast, AA or other)? Does the site have a 'Frequently Asked Questions' topic?
Comprehensiveness (C)	7	Is there information about prevention? Is there information about symptoms? Is there information about transmission? Is there diagnostic information? Is there information on where to diagnose? Is there information about the treatment? Is there information about where to take the treatment?
Readability (R)	6	Are there images or videos to support the understanding of the information? Did you have difficulty understanding prevention information? Did you have difficulty understanding information about symptoms? Did you have difficulty understanding information about transmission? Did you have difficulty understanding the diagnostic information? Did you have difficulty understanding the information about the treatment?
Accuracy (A)	20	A preventive measure is washing hands frequently with soap and water or sanitizing with 70% alcohol gel. Is this information available on this website? When coughing or sneezing, a preventive measure is covering your mouth and nose with a tissue/disposable paper or with your forearm. Is this information available on this website? A preventive measure is using a mask that covers the nose and mouth, in all environments. Is this information available on this website? A preventive measure is avoiding unnecessary circulation and maintain a minimum social distance of 1.5 m, in case it is necessary to go out. Is this information available on this website? One of the symptoms is Flu-like Syndrome characterized by at least two of the following symptoms: feverish feeling or fever, sore throat, headache, cough or runny nose, in the last 5 to 10 days. Is this information available on this website? One of the symptoms is difficulty breathing or shortness of breath (dyspnea). Is this information available on this website? One of the symptoms is loss of smell (anosmia) or decreased taste (ageusia). Is this information available on this website? One of the symptoms is tiredness and gastrointestinal symptoms (lack of appetite/nausea/vomiting/diarrhea). Is this information available on this website? Transmission takes place through one person contaminated by invisible droplets of saliva. Is this information available on this website? Transmission can be avoided with the use of a mask, hand washing and a clean, ventilated environment. Is this information available on this website? Transmission takes place through contact with surfaces such as cell phones, desks, computers and doorknobs that may be contaminated. Is this information available on this website? The transmission takes place through the household contacts of people with suspected or confirmed Covid-19, they must also observe isolation for an equal period of their cohabitants, to avoid transmission. Is this information available on this website? The diagnosis is made in a medical appointment to assess the possibility of the disease through signs and symptoms. Is this information available on this website? In cases of severe respiratory symptoms, diagnosis is performed through lung tomography. Is this information available on this website? Laboratory diagnosis is performed by collecting secretions from the nose (RTPCR) until the eighth day of onset of symptoms and/or immunological examination (rapid test or serology) indicated from the eighth day of onset of symptoms. Is this information available on this website? Diagnostic tests can be RT-PCR or immunological tests (rapid test or serology) if the individual has had contact with an infected person. Is this information available on this website? The treatment of each patient must be defined after medical evaluation depending on whether the symptoms are mild, moderate or severe. Is this information available on this website? The treatment of mild cases will be guided by doctors according to the symptoms of each patient, using analgesics, anti-allergics and medications to prevent vomiting. Is this information available on this website? The treatment of cases assessed by the doctor as moderate (shortness of breath) includes hospitalization or home monitoring. Is this information available on this website? The treatment of severe cases occurs at the hospital or emergency care unit, to maintain adequate breathing, using a mechanical ventilator, corticosteroids and/ or antibiotics and multiprofessional follow-up per the medical assessment. Is this information available on this website?

• Source: Own elaboration.

The establishment of the indicators used in the evaluation counted on 14 graduate students (12 masters and two doctoral students) from PPGDIP at UFMS, four men and 12 women. All graduated in biosciences. The two professors responsible for the discipline also participated in this activity.

The assessment was carried out during the Covid-19 pandemic. For this reason, the online completion of the evaluation tool was chosen as an alternative concerning social distancing measures to reduce the transmission of this disease. Thus, a questionnaire was produced using the Google Forms^® tool to measure the existence of information on each evaluated website. The questionnaire included a space for identifying the evaluator and the evaluated website and a set of 46 indicators. The results were stored in a spreadsheet for further analysis.

This same group carried out the evaluation process of the four websites with other external guests. Each student invited a friend or relative to make up the group of evaluators. Regarding the external guests, 13 were graduates in different areas of knowledge, and one was an undergraduate. These external guests did not undergo specific training for handling the tool and assessing. In total, we had 30 evaluators equally distributed gender-wise.

This initiative aimed to carry out a participatory evaluation of the selected websites, including people representing potential users of the information made available by the websites. Thus, we intended to address one of the shortcomings identified by Paolucci and Pereira Neto²⁵. In their systematic review, the authors found that only 2.5% of studies assessing the quality of information on health websites involved users in the evaluation process²⁵.

The websites of the State Health Secretariat of Mato Grosso do Sul (http://www.coronavirus.ms.gov.br), and of three Municipal Health Secretariats in that federation unit were selected for evaluation: Campo Grande (http://www.campogrande.ms.gov.br/sesau/covid19/), Corumbá (http://sisms.corumba.ms.gov.br/boletim/) and Dourados (http://corona.dourados.ms.gov.br). This choice was intended to present to these bodies a report containing recommendations to improve the quality of the information provided by their online environments.

Results and discussion

Table 1 presents the results obtained in assessing the quality of information on Covid-19 from four Brazilian government websites.

Table 1.
Ranking of websites on Covid-19

	Website/Criterion	T	I	R	C	A	Mean
1	Mato Grosso do Sul State Health Secretariat	72%	70%	74 %	63%	34%	62%
2	Dourados Municipal Health Secretariat	38%	70%	83%	74 %	42%	61%
3	Campo Grande Municipal Health Secretariat	37%	68%	45%	38%	23%	42%
4	Corumbá Municipal Health Secretariat	29%	31%	38%	41%	13%	30%

• Source: Own elaboration.
• T – Technical; I – Interactivity; R – Readability; C – Comprehensiveness; A – Accuracy.

Such results allow us to obtain some findings. The first evidence is related to the low quality of information identified in the four websites evaluated. Only Dourados and Mato Grosso do Sul managed to reach around 60% compliance. The others averaged below 42%.

The positive highlights of the evaluation were the interactivity and readability criteria. In the first case, the Campo Grande, Dourados and Mato Grosso do Sul websites averaged around 70%. On the Dourados and Mato Grosso do Sul websites, 80% of the evaluators identified a search tool and considered it easy to navigate on these websites. The Dourados and Mato Grosso do Sul websites obtained more than 70% compliance concerning the readability criterion. The Dourados website, for example, contains videos that facilitate the understanding of information about this disease. These results indicate that the websites of Campo Grande, Dourados, and Mato Grosso do Sul provide tools to facilitate interaction between the user and the information producer. However, it is essential to emphasize that the advantages of interactive and readable online information are reduced if the content is inaccurate, updated, and aligned with current scientific evidence²⁴,²⁵. In the accuracy criterion, all websites evaluated averaged below 42%. This rate indicates that less than half of the content available on each page evaluated complied with the information available on the official MS website dedicated to the Coronavirus.

The percentage of evaluators who are unable to find the essential information about the disease can be identified in table 2. The high rate of this percentage indicates that this information is unavailable on this site. As can be seen, in none of the cases this percentage reached 100%, which can be explained because some evaluators did not assess carefully. Also, external guests were not submitted to any training.

Table 2.
Results of accuracy indicators, by noncompliance

ACCURACY INDICATOR	PERCENTAGE OF NONCOMPLIANCE
	CG	DDOS	MS	CBA
A preventive measure is washing hands frequently with soap and water or sanitizing with 70% alcohol gel. Is this information available on this website?	53%	10%	23%	90%
When coughing or sneezing, a preventive measure is covering your mouth and nose with a tissue/disposable paper or with your forearm. Is this information available on this website?	50%	7%	13%	87%
A preventive measure is using a mask that covers the nose and mouth, in all environments. Is this information available on this website?	77%	43%	70%	83%
A preventive measure is avoiding unnecessary circulation and maintain a minimum social distance of 1.5 m, in case it is necessary to go out. Is this information available on this website?	77%	73%	53%	83%
One of the symptoms is Flu-like Syndrome characterized by at least two of the following symptoms: feverish feeling or fever, sore throat, headache, cough or runny nose, in the last 5 to 10 days. Is this information available on this website?	80%	60%	73%	77%
One of the symptoms is difficulty breathing or shortness of breath (dyspnea). Is this information available on this website?	70%	17%	3%	57%
One of the symptoms is loss of smell (anosmia) or decreased taste (ageusia). Is this information available on this website?	80%	90%	93%	93%
One of the symptoms is tiredness and gastrointestinal symptoms (lack of appetite/nausea/vomiting/diarrhea). Is this information available on this website?	83%	87%	93%	93%
Transmission takes place through one person contaminated by invisible droplets of saliva. Is this information available on this website?	77%	17%	23%	93%
Transmission can be avoided with the use of a mask, hand washing and a clean, ventilated environment. Is this information available on this website?	77%	40%	63%	90%
Transmission takes place through contact with surfaces such as cell phones, desks, computers and doorknobs that may be contaminated. Is this information available on this website?	73%	40%	43%	93%
The transmission takes place through the household contacts of people with suspected or confirmed Covid-19, they must also observe isolation for an equal period of their cohabitants, to avoid transmission. Is this information available on this website?	87%	70%	83%	67%
The diagnosis is made in a medical appointment to assess the possibility of the disease through signs and symptoms. Is this information available on this website?	83%	80%	83%	90%
In cases of severe respiratory symptoms, diagnosis is performed through lung tomography. Is this information available on this website?	93%	90%	90%	93%
Laboratory diagnosis is performed by collecting secretions from the nose (RT-PCR) until the eighth day of onset of symptoms and/or immunological examination (rapid test or serology) indicated from the eighth day of onset of symptoms. Is this information available on this website?	73%	73%	87%	93%
Diagnostic tests can be RT-PCR or immunological tests (rapid test or serology) if the individual has had contact with an infected person. Is this information available on this website?	83%	73%	83%	93%
The treatment of each patient must be defined after medical evaluation depending on whether the symptoms are mild, moderate or severe. Is this information available on this website?	70%	70%	80%	93%
The treatment of mild cases will be guided by doctors according to the symptoms of each patient, using analgesics, anti-allergics and medications to prevent vomiting. Is this information available on this website?	90%	67%	83%	93%
The treatment of cases assessed by the doctor as moderate (shortness of breath) includes hospitalization or home monitoring. Is this information available on this website?	77%	73%	83%	93%
The treatment of severe cases occurs at the hospital or emergency care unit, to maintain adequate breathing, using a mechanical ventilator, corticosteroids and/or antibiotics and multiprofessional follow-up per the medical assessment. Is this information available on this website?	90%	83%	10%	93%
Mean	77%	58%	62%	87%

• Source: Own elaboration.
• MS (Mato Grosso do Sul State Health Secretariat); DDOS (Dourados Municipal Health Secretariat); CG (Campo Grande Municipal Health Secretariat); CBA (Corumbá Municipal Health Secretariat).

The results presented in table 2 seem troubling. Regarding prevention, for example, mask use is one of the preventive measures recognized and recommended by the WHO to curb the spread of Covid-19²⁷. However, this information was not available or was difficult to see on the websites of Corumbá, Campo Grande, and the State Secretariat of Mato Grosso do Sul. In the case of these virtual environments, attention was focused on providing epidemiological information on the evolution of the pandemic in these regions of the state instead of providing information on disease prevention. On the Corumbá website, there is no information related to the necessary frequent hand hygiene or the use of 70% alcohol gel.

We noticed that most of the essential information regarding transmission is missing. Noteworthy is that the Campo Grande and Corumbá websites fail to report that the transmission occurs through contact with surfaces such as cell phones, tables, computers, and doorknobs that may be contaminated. None of the four websites report that transmission can occur through the household contacts of people with suspected or confirmed Covid-19.

We also observed that none of the websites evaluated contained minimum necessary diagnostic information. Almost all evaluators did not identify information regarding the diagnostic test, the need for a medical visit to assess the possibility of having the disease, and the performance of tomography in the case of severe respiratory symptoms.

Disease treatment indicators also showed low compliance in all websites evaluated. Almost all evaluators found that information on this dimension was absent or hardly seen in the four digital environments evaluated.

Some aspects deserve a brief discussion given the results presented above. One of them is related to the methodological procedure adopted in the research that became this paper. As mentioned, the research participants carried out the readability assessment of the four websites.

Eysenbach et al.²⁴ had already shown that “none of the studies conducted comprehension tests with actual consumers or used judgments of literacy experts”^24(2695). Also, they emphasized that the studies used technological resources to assess the readability of information. These authors argue that such formulas

do not reflect other factors that affect comprehension such as frequency and explanation of medical jargon, writing style [...] or use of culturally specific information^24(2694).

In his systematic review, Paolucci²⁵ concluded that the readability of a website was still performed using formulas such as Flesh-Kincaid Grade Level, Discern tool, and Flesch Reading Ease. Thus, this paper dialogues with the international literature and proposes involving the end-user in the readability assessment.

Another aspect related to the methodological dimension refers to the criteria and indicators used. In this case, the evaluation model adopted in this work differs, in many aspects, from two others present in the international academic scene. One of them is the Health On Net Foundation (HON). Another is the Discern Questionnaire (DQ).

HON (https://www.hon.ch/en/) is a nongovernmental organization created in 1995, based in Switzerland. It evaluates health pages based on eight principles, namely: ‘authorship’, which is related to the presence of the source of the information offered; ‘complementarity’, which is concerned with the fact that the information available does not replace the doctor-patient relationship; ‘confidentiality’, which assesses whether the information provided by patients has not been made public; ‘attribution and update’, which seeks to verify whether the last date of insertion of the information on the website is included; ‘guarantee’, refers to the fact that the digital environment reveals that it has no commercial interest; ‘transparency’ is concerned with users’ access to websites through the several internal communication tools (contact us, e-mail, Facebook); ‘sponsorship transparency’ is dedicated to identifying whether the website indicates its sources of funding; and ‘honesty and transparency about advertising’²⁴,²⁵. While some of the criteria adopted in this paper coincide with the principles adopted by the HON, no reference is made to the accuracy and legibility of information among the principles adopted by the Swiss institution.

The DQ (http://www.discern.org.uk/index.php) is a project founded in 1996-1997 by the British Library and the National Health Services (NHS). This questionnaire provides users with means to assess the quality of written information about treatment options for a health problem. It is divided into three sections. The first is concerned with information reliability. The second leads users to question the quality of information about treatment options. The third guides the user to carry out a general classification of the publication. One of the questions, for example, asks whether the website ‘describes how each treatment works’. In this case, the website may describe this aspect in an incorrect, obsolete, or incomprehensible way. The DQ²⁴,²⁵ does not consider this aspect.

For this reason, the criteria and evaluation process adopted in the research that resulted in this paper may be complete. Another point that deserves to be highlighted in the discussion of this paper can be highlighted when comparing the results obtained in this research with others found in previous studies. The problem of low informational quality on institutional websites linked to health secretariats had already been observed in two previous studies: one evaluating the information on dengue websites²⁸, and another checking the quality of information on tuberculosis websites²⁹.

Pereira Neto et al.²⁸ assessed the quality of information on 20 websites about dengue, using the same criteria and following a participatory perspective similar to that carried out in the research that resulted in this paper. Seven were linked to Brazilian government agencies, namely: The Ministry of Health (MS); the Oswaldo Cruz Institute (IOC), a unit of the Oswaldo Cruz Foundation; the Dengue Network of the Oswaldo Cruz Foundation (RD); the State Health Secretariats of Paraná (SES-PA), Rio de Janeiro (SES-RJ), and Minas Gerais (SES-MG); and the Recife Municipal Health Secretariat (SMS-RE).

Paolucci, Pereira Neto, and Luzia²⁹ assessed the quality of information on 12 websites about tuberculosis, using the criteria and perspective of participant research adopted in the study presented here. Six of the websites evaluated²⁹ are linked to Brazilian government agencies, namely: National Tuberculosis Control Program; MS; Tropical Medicine Foundation; Amazonas (AM) state government; SES-PA and São Paulo (SES-SP); and Municipal Health Secretariats of Porto Alegre (SMS-PA) and Rio de Janeiro (SMS-RJ). The results obtained in the evaluation of information on Brazilian government websites about dengue²⁸ were as follows:

Table 3.
Results of the evaluation of government websites on dengue

Website/Criterion	Technical	Interactivity	Comprehensiveness	Accuracy	Readability	Total
RJ State Health Secretariat	41,6%	89,5%	66,9%	65,7%	46,8%	62,1%
PR State Health Secretariat	62,7%	54,0%	64,1%	38,6%	45,8%	53,0%
Ministry of Health	37,5%	76,0%	57,8%	62,9%	28,4%	52,5%
MG State Health Secretariat	28,6%	67,5%	58,1%	20,7%	42,8%	43,6%
Rede Dengue (Fiocruz)	48,2%	61,0%	36,9%	24,3%	29,2%	39,9%
Recife Municipal Health Secretariat	23,9%	25,0%	50,9%	23,6%	33,8%	31,4%
Oswaldo Cruz Institute	36,4%	24,0%	23,1%	11,4%	16,8%	22,3%

• Source: Pereira Neto et al.²⁸.

The results obtained in the evaluation of information on Brazilian government websites on tuberculosis²⁹ were as follows:

Table 4.
Results of the evaluation of government websites on tuberculosis

Website/Criterion	Technical	Interactivity	Comprehensiveness	Readability	Accuracy	Total
SP State Health Secretariat	44%	77%	83%	41%	72%	63%
PNCT	38%	76%	74 %	41%	74 %	61%
RJ Municipality	52%	79%	66%	40%	47%	57%
Porto Alegre Municipality	30%	76%	67%	38%	50%	52%
PR State Health Secretariat	41%	50%	56%	29%	45%	44%
Fund. Med. Tropical AM	33%	23%	54%	18%	34%	32%

• Source: Paolucci, Pereira Neto, and Luzia²⁹.
• PNCT – National Tuberculosis Control Program.

The compliance rates identified in evaluating Brazilian government websites about dengue are noteworthy. Only one of them had an overall average of over 60% compliance. Regarding accuracy, only two had more than 60% compliance²⁹. Regarding tuberculosis, Paolucci, Pereira Neto, and Luzia²⁹ found that the highest mean of compliance was around 60%. In terms of accuracy, only one achieved more than 70% compliance.

The compliance percentages identified in the study that resulted in this paper reveal that the quality of information about Covid-19 in the Brazilian government websites evaluated is poor. In general terms, this result is close to that obtained by Pereira

Neto et al.²⁸ and Paolucci, Pereira Neto, and Luzia²⁹ in the evaluations of dengue and tuberculosis websites, respectively. The data obtained in the study presented here and those available in the literature²⁸,²⁹ led to affirm that Brazilian government websites must be evaluated and then modified, considering the minimum conditions of interactivity, readability, and accuracy.

Final considerations

Online health information can be an essential resource for increasing self-care, self-management, empowerment, and adherence to treatment. Therefore, it should have quality and must be interactive, understandable, current, reliable, and scientifically proven²⁴. We should stress that it did not refer to its subjective dimension when using the word ‘quality’. The researchers in this study believe that this expression, associated with online health information, is related to the perspective of survival and well-being that it can provide to the population, which can be measured by criteria and indicators³⁰.

The Pan American Health Organization has just confirmed the existence of infodemia, that is, “excess information, some accurate and some not, which hinder finding reputable sources and reliable guidance when needed”³¹⁽²⁾.

What has been done to relieve this problem?

Last year, the consortium formed by institutions linked to the Ministry of Science, Technology, Innovation, and Communications (MCTIC) and the MS launched a call to

support research projects that aim to significantly contribute to the country’s scientific and technological development and innovation in coping with Covid-19, its consequences, and other severe acute respiratory syndromes³².

One of the themes for the submission of proposals was prevention and control. It is subdivided into five lines. One of them predicted:

Support the development of systematic reviews, evidence syntheses, or qualitative studies on good communication strategies and prevention of fake news circulation to the population, including approaches to assess fear, stigma, anxiety, hearsay related to Covid-19 and other severe acute respiratory syndromes³²⁽³⁾.

We an observe an interest from the leading public funding agencies in supporting research on the topic of fake news during the Covid-19 pandemic. This paper revealed another severe issue. Brazilian government websites have shortcomings and fail to meet the minimum quality criteria, with incorrect, obsolete, incomplete, and illegible data on the subject.

The websites evaluated showed that, in general, the level of compliance with the criteria and indicators used was equal to or less than 61%. In the accuracy criterion, the percentages obtained were below 45%. Thus, we can observe that the online environments of the health secretariats dedicated to Covid-19 do not meet the minimum quality criteria.

In a context in which increasingly more people use the internet to search for health information, the content made available, especially on the websites of Brazilian government agencies, should be technical, comprehensive, interactive, readable, and accurate. We should emphasize that, by achieving the indicated compliance, within the quality of information, the website will be more accessible to citizens, increasing trust in the information offered and, above all, contributing to the circulation of information based on scientific and for coping with fake informational content.

The assessment and certification of Brazilian government websites are an essential alternative to combat fake news on health, especially in a pandemic context. Furthermore, public websites should strive to provide updated, correct, understandable, and quality information. Unfortunately, this was not the reality found due to this assessment. If the public website is of quality, Brazilians will have a place to turn to when they have any doubts regarding information about Covid-19. Brazil has no public or private agency developing systematic activities to assess the quality of health information available on the internet.

Meanwhile, in the United States and Europe, government agencies, professional associations, and researchers have been evaluating the information on health websites for more than two decades. The HON and DQ are the most prominent. Wouldn’t it be the case to build a public agency certifying government health websites?

Limits and potentials

It is noteworthy that the study conducted used a small sample of websites for analysis. However, this evaluation proposal based on criteria and indicators built in a participatory way from systematic reviews can be used to evaluate a more significant number of websites from different Brazilian cities and states. Thus, it will be possible to offer a more comprehensive overview of the quality of information about Covid-19 on websites of public bodies nationwide.

Participatory evaluation is a feature that deserves to be highlighted in the experience carried out and transformed in this paper. Participation occurred at all stages of the research and health website certification processes. Participants contributed to the construction and validation of the indicators and the evaluations of the websites. It is noteworthy that most evaluators reside in Mato Grosso do Sul and have higher education. Thus, while it was not a selection condition, it is reasonable to assume they had health literacy and a similar socioeconomic status. This aspect limits the possibility of generalizing the findings. Despite this, this paper evidences some troubling results.

The high circulation of fake news has marked the Covid-19 pandemic. In this context, the webpages of health secretariats should show high quality and readability information about the disease. Only then will they offer reliable, informative content based on scientific evidence, contributing to coping with fake news and its negative impacts.

This paper aims to encourage further research on the subject. In this sense, Paolucci’s doctoral thesis³³ presents an innovative perspective that significantly contributes to the academic debate on the subject. It also aims to include the theme in the research agenda of the MS and the Ministry of Science and Technology and can sensitize health policymakers about the problem of quality of information and the consequences of a quality website for the well-being of citizens. Further studies that identify the short and long-term effects that this information can promote, including health costs¹⁰, would be very valuable. Studies evaluating the quality of information on diseases prevalent in developing countries are also noteworthy.

In this paper, we sought to highlight the relevance of the problem of the quality of health information available on the internet, particularly on Brazilian government websites on the Covid-19 pandemic. He suggests other possibilities to reduce this problem, such as health literacy and the creation of news checking agencies.

As presented in this work, the topic has been receiving academia’s attention for some time. In this sense, one of the questions raised by this paper is ‘what can be done so that the issue of evaluating the quality of health websites starts to be discussed by society and becomes a relevant point in the research agenda of development agencies?’.

• Financial support: non-existent

Referências

• ¹
  Rice R, Yates S, Blejmar J. Introduction to the Oxford Handbook of digital technology and society: terms, domains, and themes. In: Yates S, Rice R, organizadores. The Oxford Handbook of digital technology and society. Oxford University Press: New York; 2020. p. 3-35.
• ²
  Pereira Neto A, Lima J, Barbosa L, et al. Internet, expert patient e empoderamento: perfis de atuação em comunidades virtuais de renais crônicos. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 145-175.
• ³
  Pereira Neto A, Paolucci R. Avaliação da qualidade da informação de saúde na internet: análise das iniciativas brasileiras. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 257-291.
• ⁴
  World Health Organization. Coronavirus disease (COVID-19) pandemic. Genebra: WHO; 2020 [acesso em 2021 abr 9]. Disponível em: https://www.who.int/emergencies/diseases/novel-coronavirus-2019
  » https://www.who.int/emergencies/diseases/novel-coronavirus-2019.
• ⁵
  Fundação Oswaldo Cruz. Monitora Covid-19. Rio de Janeiro: Fundação Oswaldo Cruz; 2020. [acesso em 2021 abr 9]. Disponível em: https://bigdata-covid19.icict.fiocruz.br/
  » https://bigdata-covid19.icict.fiocruz.br/.
• ⁶
  Sharifi A, Khavarian-Garmsir A. The COVID-19 pandemic: Impacts on cities and major lessons for urban planning, design, and management. Sci Total Environ. 2020; (749):1-14.
• ⁷
  Santos RC, Carvalho TA, Souza Neto IF, et al. Urgency and emergency in times of covid-19 – an integrative literature review. Res., Soc. Dev. 2021; 10(3):e9110313027.
• ⁸
  Silva D, Cobucci RN, Soares-Rachetti VP, et al. Prevalência de ansiedade em profissionais da saúde em tempos de covid-19: revisão sistemática com metanálise. Ciênc. Saúde Colet. 2021; 26(2): 693-710.
• ⁹
  Diniz MC, Martins M, Xavier K, et al. Crise global coronavírus: monitoramento e impactos. Cadernos de Prospecção. 2020; 13(2):359-377.
• ¹⁰
  Spoelman WA, Bonten TN, Waal MW, et al. Effect of an evidence-based website on healthcare usage: an interrupted time-series study. BMJ Open. 2016; 6(11).
• ¹¹
  Lagan BM, Sinclair M, Kernohan WG. What Is the Impact of the Internet on Decision-Making in Pregnancy? A Global Study. Birth. 2011; 38(4):336-345.
• ¹²
  Garbin H, Guilam M, Pereira Neto A. Internet na promoção da saúde: um instrumento para o desenvolvimento de habilidades pessoais e sociais. Physis. 2012; 22(1):347-363.
• ¹³
  Lima DL, Lopes M, Brito AM. Social media: friend or foe in the COVID-19 pandemic?. Clinics. 2020; (75):e1953.
• ¹⁴
  Gonzalez-Padilla DA, Tortolero-Blanco L. Social media influence in the COVID-19 Pandemic. Int. Braz. J. Urol. 2020; (46):120-124.
• ¹⁵
  Ho H, Chen Y, Yen C. Different impacts of COVID-19-related information sources on public worry: An online survey through social media. Internet Intervs. 2020; (22):1-6.
• ¹⁶
  Barbosa L. Comunicação e informação sobre câncer de mama: um estudo comparativo entre um ambiente físico e outro on-line. [tese]. Rio de Janeiro: Fundação Oswaldo Cruz; 2021.
• ¹⁷
  Fiedorowicz JG. New challenges in the COVID-19 pandemic. J. Psyc. Research. 2020; (133):110123.
• ¹⁸
  Pereira Neto A, Barbosa L, Flynn M. Prefácio da edição brasileira – Há décadas em que nada acontece. Há semanas em que décadas acontecem. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 7-18.
• ¹⁹
  Vasconcellos-Silva PR, Castiel LD. COVID-19, as fake news e o sono da razão comunicativa gerando monstros: a narrativa dos riscos e os riscos das narrativas. Cad. Saúde Pública. 2020; 36(7):e00101920.
• ²⁰
  Galhardi CP, Freire N, Minayo M, et al. Fato ou Fake? Uma análise da desinformação frente à pandemia da Covid-19 no Brasil. Ciênc. Saúde Colet. 2020; (25):4201-4210.
• ²¹
  Fonseca VPS, Freitas F, Gehrke M, et al. Jornalismo guiado por dados como ferramenta de fact-checking: uma experiência laboratorial. Comum. Inov. 2018; (19):67-83.
• ²²
  Souza N. eHealth literacy entre jovens: estudo exploratório sobre o papel das condições socioeconômicas no uso da informação sobre saúde na Internet. [dissertação]. Rio de Janeiro: Fundação Oswaldo Cruz; 2020.
• ²³
  Sousa RA, Moreira K, Farinelli M, et al. Literacia Digital e interfaces com a saúde: uma revisão integrativa. In: Pereira Neto A, Flynn M, organizadores. Internet e saúde no Brasil: desafios e tendências. São Paulo: Cultura Acadêmica; 2021. p. 421-443.
• ²⁴
  Eysenbach G, Powell J, Kuss O, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama. 2002; 287(20):2691-2700.37.
• ²⁵
  Paolucci R, Pereira Neto A. Methods for evaluating the quality of information on health websites: Systematic Review (2001-2014). Latin Am J Develop. 2021; 3(3):994-1056.
• ²⁶
  Brasil. Lei nº 13.146, de 6 de julho de 2015. Estatuto da Pessoa com Deficiência. Diário Oficial da União. 7 Jul 2015.
• ²⁷
  World Health Organization. Mask use in the context of Covid-19. Genebra: WHO; 2020 [acesso em 2021 abr 9]. Disponível em: https://www.who.int/publications/i/item/advice-on-the-use-of-masks--in-the-community-during-home-care-and-in-he-althcare-settings-in-the-context-of-the-novel-co-ronavirus-(2019-ncov)-outbreak
  » https://www.who.int/publications/i/item/advice-on-the-use-of-masks--in-the-community-during-home-care-and-in-he-althcare-settings-in-the-context-of-the-novel-co-ronavirus-(2019-ncov)-outbreak
• ²⁸
  Pereira Neto AF, Paolucci R, Daumas RP, et al. Avaliação participativa da qualidade da informação de saúde na internet: o caso de sites de dengue. Ciênc. Saúde Colet. 2017; 22(6):1955-1968.
• ²⁹
  Paolucci R, Pereira Neto A, Luzia R. Avaliação da qualidade da informação em sites de tuberculose: análise de uma experiência participativa. Saúde debate. 2017; 41(esp):84-100.
• ³⁰
  Paim I, Nehmy RMQ, Guimarães CG. Problematização do conceito ‘qualidade’ da Informação. Perspec. Ci. Inf. 1996; 1(1):111-119.
• ³¹
  Organização Pan-Americana da Saúde. Entenda a infodemia e a desinformação na luta contra a Covid-19. Brasil: OPAS; 2020. [acesso em 2021 abr 9]. Disponível em: https://iris.paho.org/bitstream/handle/10665.2/52054/Factsheet-Infodemic_por. pdf?sequence=14
  » https://iris.paho.org/bitstream/handle/10665.2/52054/Factsheet-Infodemic_por. pdf?sequence=14.
• ³²
  Brasil. Ministério da Ciência Tecnologia, Inovações e Comunicações, Conselho Nacional de Desenvolvimento Científico e Tecnológico; Ministério da Saúde. Chamada MCTIC/CNPq/FNDCT/MS/SCTIE/Decit Nº 07/2020 – Pesquisas para enfrentamento da COVID-19, suas consequências e outras síndromes respiratórias agudas graves. Brasília, DF: Ministério da Ciência Tecnologia, Inovações e Comunicações; 2020. [acesso em 2021 abr 9]. Disponível em: https://www.gov.br/cgu/pt-br/governo-aberto/noticias/2020/4/governo-federal-lanca-chamada-de-apoio-a-pesquisas-em-coronavirus
  » https://www.gov.br/cgu/pt-br/governo-aberto/noticias/2020/4/governo-federal-lanca-chamada-de-apoio-a-pesquisas-em-coronavirus
• ³³
  Paolucci R. Avaliação da qualidade da informação em sites de saúde: indicadores de acurácia baseada em evidência para tuberculose. [tese]. Rio de Janeiro: Fundação Oswaldo Cruz; 2020.

Publication Dates

• Publication in this collection
  21 Feb 2022
• Date of issue
  Jan-Mar 2022

History

• Received
  15 Apr 2021
• Accepted
  29 Nov 2021