COVID-19 pandemic information on Brazilian websites: credibility, coverage, and agreement with World Health Organization

OBJECTIVE: To assess the credibility and the quality content of COVID-19 pandemic information on Brazilian websites. METHODS: We performed Google searches and screened the first 45 websites. The websites were categorized as academic, commercial, government, hospital, media, nongovernmental organizations, and professionals. The credibility was assessed by JAMA benchmark criteria and HONCODE. A checklist with WHO information about COVID-19 was developed to assess the quality content. For each website, the level of agreement with WHO information was categorized into “total,” “partial,” or “disagreement”. RESULTS: A total of 20 websites were analyzed. None of the websites had HONCODE certification. Six websites (30%) met none of the four JAMA criteria and only one website (5%) fulfilled all the four criteria. Only 11 out of 20 websites showed overall coverage >50% for the checklist. Overall, 70% (14/20) of the websites had at least 50% total agreement with WHO items. The government websites presented more disagreement with the WHO items than media websites in the overall quality content analysis. CONCLUSION: The COVID-19 information on Brazilian websites have a moderate-to-low credibility and quality, particularly on the government websites.


INTRODUCTION
In December 2019, the Chinese National Council and the World Health Organization (WHO) were both notified about the first cases of patients with pneumonia from an unknown source and shared symptoms in the city of Wuhan, China. It was discovered that these symptoms were caused by the SARS-CoV-2, a novel coronavirus, and the disease developed by this coronavirus was called COVID-19. The accelerated spreading of the COVID-19 outbreak led the WHO to announce that it became pandemic in March 2020 [1][2][3] .
The COVID-19 pandemic is one of the biggest concerns of the 21st century. As less information is available about this novel disease, the scientific community is increasingly engaged in understanding the most important parameters to manage the disease and to reduce the spreading of the outbreak. As a result, emerging information is quickly shared by researchers, international organizations, and media 3 .
Nowadays, the Internet has been a common source of health information to the population. However, as any Internet user can provide content on the Internet regardless the quality or accuracy of the content, a lot of fake news related to health issues have been shared in the social media, blogs, and websites. During a pandemic, the fake news may impair the prevention, spreading, and treatment of the disease 4 . Unfortunately, it is difficult for the population in general to distinguish the reliable information and the fake news on the Internet once people are not critical about the content that is consumed by them.
According to the National Household Sample Survey, the Internet utilization in households increased from 74.9% to 79.1% within 1 year 5 . This increased use of Internet is especially important because the individuals have valued more Internet-based information than the information derived from the elders of the family, as it was in the past 6 . Given that the main sources of online information are websites and blogs and that the most searched content during a global outbreak involves the prevention and the treatment of the new disease, reliable online information is crucial to improve the pandemic-related health outcomes 7,8 .
To provide accurate information about the definition, symptoms, spreading, prevention, treatment, and other aspects related to the new disease COVID-19, the WHO has released reports covering public advice, country and technical guidance, frequently asked questions, travel advice, and mythbusters 9 . Considering that part of the Internet-based information is inaccurate, the aim of this study was to assess the quality and credibility of COVID-19related information available in websites and to verify the agreement of that information with the WHO reports and advices.

METHODS
This qualitative study was conducted between April and May 2020. Online searches were performed by using the advanced Google search engine in the Google Chrome browser with the search terms "coronavirus" and "COVID-19." The searches were limited to Brazilian websites published in Portuguese. All browser caches and cookies were cleared before searching. The first 45 websites were screened for analysis. The website would contain a specific webpage for information related to the COVID-19 aimed to the general population. The websites containing only daily news or information directed toward a specific public, or redirecting to obtain the main information in other websites were excluded. As adopted in the previous studies, up to four webpages of each website were considered for this analysis [10][11][12] . The searches, the screening for inclusion and exclusion, the categorization of the websites, and the assessment of the outcomes were all performed by two independent researchers. Any discrepancy was resolved by consensus.
Each website was classified into one of the following categories: academic (from universities, schools, or educational channels), commercial (with commercial purposes), governmental (from government agencies), media (from media corporations), professional (from health professionals with or without academic credentials), hospital (from medical centers, clinics, and hospitals), and nongovernmental organizations (from NGOs).
The outcomes were the credibility and the content quality and coverage of the websites. The HONCODE and JAMA benchmark criteria were used to assess the website credibility. The quality and coverage of website contents were evaluated by using a qualitative content analysis. HONCODE is a system created in 1995 that provides an electronic certification to high-standard websites 13 . To have the HONCODE certification, a website may follow eight ethics principles: authority, complementarity, confidentiality, attribution, justifiability, transparency, financial disclosure, and advertising 4 . We have checked whether the websites were certified by HONCODE through the HON website http://www. healthonnet.org/. Each website was categorized into "yes" (certified) or "no" (not certified).
The JAMA benchmark criteria is one of the most common tools used to assess the quality of Internet-based information. It assesses four components: authorship (whether the website states the authors and contributors), attribution (whether the website states the references and sources), disclosure (whether the website discloses any conflict of interest, sponsorship, advertising, etc.), and currency (whether the website states dates and update dates) 4,14 . Each website was categorized into "yes" or "no" for each criterion.
In order to assess the quality and coverage of the website contents, we developed a 57-item checklist with WHO information related to the COVID-19 pandemic. These 57 items were divided into definition (2/57), symptoms (4/57), spreading (9/57), prevention (17/57), treatment (7/57), and others (18/57). For quality assessment, we measured the agreement between the website information and each item from WHO checklist. The agreement was judged to be "total" when website information exactly matched the WHO information, "partial" when website information were incomplete in comparison with the WHO information, and "disagreement" when website information were totally discordant to the WHO information. For website coverage evaluation, we measured using the 57-item WHO checklist. For each item covered by the website, one point was attributed. If the website contains more content related to WHO information, more points were provided.

RESULTS
From 45 websites screened for eligibility, 25 were excluded. The reasons for exclusion were: not having a specific webpage for COVID-19-related information (19 websites), redirecting to other websites for information (2 websites), having information directed at a specific audience (3 websites), and not a Brazilian website (1 website). From the 20 included websites, 8 websites were authored by media corporations, 6 websites by governmental agencies, 3 websites by hospitals, 1 website by commercial entities, 1 website by professionals, and 1 website by academic sources.
None of the websites had HONCODE certification. Six websites (30%) met none of the four JAMA benchmark criteria. Three websites (15%) met one JAMA criteria, four websites (20%) met two JAMA criteria, six websites (30%) met three JAMA criteria, and only one website (5%) fulfilled all the four JAMA benchmark criteria ( Table 1).

DISCUSSION
Although Google does not have a strong control of quality content, it is the most used search engine in the world 4,11,15 . For this reason, we have used Google engine to fetch the websites presenting COVID-19 information. As most Internet users do not go beyond the second page of web search, we have only screened the first 45 websites in order to include at least 15 websites for the final analysis 15 . A total of 25 websites were excluded mainly because of reporting only the daily news about the pandemic situation in the country and not having a specific webpage to provide information related to the main aspects of the COVID-19 disease.
The WHO has been crucial to guide governmental actions around the world during the COVID-19 pandemic. This international organization has provided updated, quick, and reliable information to avoid misunderstandings 9 . Our findings show that although 75% of the Brazilian websites had total agreement with at least half of the WHO items related to prevention, none of the websites exhibited agreement with all the WHO prevention items. Many websites had incomplete or mistaken information related to the minimum recommended distance among people, the need to avoid crowded places and travels, and the recommendation about stay home when people feel unwell. These findings are important because studies have showed that quarantine, social distancing, and isolation are helpful to reduce COVID-19 outbreak 9,16 . Therefore, the websites should provide accurate information to the population about these prevention strategies.
We have found that the government websites presented more disagreement with WHO items than media websites in the overall quality content analysis. Considering that WHO is an international public health agency that provides updated outbreak situation reports, builds research partnerships to accelerate the development of vaccines and drugs, and has handled other pandemics in the past, governments should be aligned with the WHO guidance to ensure a better management of COVID-19 pandemic 17 . The observed disagreement between WHO and government websites (from 8.6% to 33.3%) highlights the need for improvement in health communication.
During a pandemic, effective health communication plays an essential role on reducing the fear and anxiety related to the outbreak and increasing the adherence to adequate prevention measures 18 . Positively, Brazilian media websites showed high quality content and less disagreement with WHO information (from 0% to 8.5%). It is especially relevant because media have a large audience on Internet. The major four Brazilian media companies reach approximately 58.7% of the digital population 19 . Therefore, COVID-19 information provided by media websites is expected to have a strong quality as information is quickly disseminated and reaches a huge number of people.
For the JAMA benchmark analysis, out of six websites that met none of the criteria, four websites appeared either on the first or on the second page of Google search, and five of them were authored by government agencies. It is intriguing because the websites with less credibility are located in the most viewed pages of web searches and are created by the government, which should be the most reliable source of information as people tend to follow the government instructions   Table 3. Quality content analysis of the individual websites.
Note: The level of websites in agreement with the WHO items is expressed overall and per categories.
in the crisis management 16 . The most credible websites were authored by media. These findings are similar to the aforementioned quality content analysis, where media websites showed higher quality than the government websites. Website coverage of WHO information was low. Only half of the websites presented at least 50% coverage of items from WHO checklist. The lack of reliable information can lead to serious negative effects on the population such as the panic, excessive buying of supplies such as food and hygiene items, and unnecessary medications without medical prescription, which affects people especially those suffering from chronic diseases 20 . Therefore, accurate information may be freely and easily available to overall population to stimulate adequate behaviors and to keep population aware of the better ways of preventing and managing the COVID-19 outbreak.
This study has some limitations. First, the assessment of quality content was based on the agreement between website information and WHO guidelines and recommendations. As the Brazilian Ministry of Health has some divergent recommendations from WHO, some websites followed Ministry of Health guidelines and consequently did not agree with the WHO items. Second, the WHO checklist used to verify the agreement with the websites was developed by our research group. Therefore, it may not be considered as a valid tool. Third, information related to a pandemic change quickly, which just allows to assess the information at a specific point in time.

CONCLUSIONS
Our findings showed that Brazilian websites have poor-to-moderate credibility and quality of content regarding the COVID-19 pandemic, while compared with the WHO information items. Furthermore, it was observed that the government websites are less credible and have lower quality content than media websites. These findings highlight the need for increasing the reliability of online health information about COVID-19 pandemic and developing the strategies to turn Internet users able to identify the quality and credibility of the online health information they are consuming.  Table 3:

AUTHORS' CONTRIBUTIONS
Where it reads: