Usefulness, Reliability, Quality, and Content of Portuguese Language YouTube Videos on COVID-19

Abstract

Objective:

To investigate the usefulness, reliability, quality, and content of Portuguese-language YouTube videos on COVID-19.

Material and Methods:

Three keywords selected on Google Trends were searched on YouTube, and the 60 first videos listed with each term were analyzed. Two calibrated researchers evaluated the reliability (DISCERN Modified Scale), the quality (Global Quality Score - GQS), and the usefulness of videos for the users (COVID-19 Specific Score - CSS). The number of views, likes, and engagement were also analyzed. The data were evaluated by the Mann-Whitney test and Spearman correlation (α=5%).

Results:

59 videos were included. The average scores of quality, reliability, and usefulness were 3.0 (±1.1), 3.2 (± 0.8), and 1.5 (± 0.9), respectively. Two-thirds of the videos (64.4%) had low/moderate quality, and the majority (76.2%) were about signs and symptoms. The numbers of views (p=0.005), likes (p=0.006), and engagement (p=0.014) were significantly higher in moderate/good quality videos. The number of comments (p=0.007), duration of videos (p=0.004), and the DISCERN score (p<0.001) were significantly higher in videos made by health professionals. The general quality of the videos was positively correlated to the CSS scores, number of views, likes, and engagement (p<0.05).

Conclusion:

Most videos had moderate quality and reliability and low usefulness for the users.

Keywords:
SARS-CoV-2; COVID-19; Pandemics; Virus Disease; Internet-Based Intervention

Introduction

Society today is known to be connected and privileged by the excellent condition to get information and knowledge due to the internet. In this context, people use the internet not only for entertainment but also for medical information [¹]. It is estimated that 8 out of 10 internet users search for information about health [²]. The internet often outperforms traditional media such as radio, TV, and print regarding health information [³]. Video platforms are the most used media, and in pandemic times, this tool may be even more used [⁴].

Youtube™ is a digital video-sharing tool widely used to obtain knowledge. According to internet traffic data, Youtube™ has been the second most visited website in the world since April 2017, surpassed only by the Google search engine [³]. However, this platform is not entirely reliable once anyone creates content and spreads news not checked by any veracity verification process [²,⁵]. Thus, the published content may spread fake news, starting public paranoia and panic and making it difficult to control, especially during the pandemic [³]. Previous studies that evaluated the quality and reliability of educational videos published on this platform found many videos showing fake or unuseful news [²,³,⁵,⁶].

COVID-19, caused by the virus SARS-CoV-2, was defined as a pandemic by the World Health Organization in March 2020, and it has been responsible for more than 5 million deaths in the world up to January 2022 [⁷] It is a new virus transmitted from person to person through the respiratory tract, through proximity that results in the inoculation of the virus through the mouth, nose, or eye and contact with objects used by infected people [⁸]. At the same time that YouTube™ can be a great ally in disseminating useful information, it can, when used inadvertently, spread false news that discourages compliance with biosecurity measures, contributing significantly to the increased risk of contagion and generating mistrust about the vaccine [⁸]. A previous study on the quality and reliability of English-language YouTube videos on COVID-19 found that approximately a quarter of the videos needed to have corrected information [¹⁰]. Another study, also concerning English-language videos, showed that YouTube videos about pregnancy and COVID-19 had low reliability and quality [¹¹]. However, studies have yet to be carried out on the videos available in Portuguese. Thus, this study aimed to evaluate the usefulness, reliability, quality, and content of educational Portuguese-language YouTube™ videos on COVID-19.

Material and Methods

Study Design

An observational and infodemiological study was conducted by analyzing educational videos about COVID-19 on the Youtube™ platform.

Videos Search and Selection Strategy

Videos were selected with the aid of Google Trends (https://trends.google.com.br/trends/?geo=BR), a website that shows the most searched terms in the recent past. The three most searched terms were: “coronavirus”, “covid 19”, “covid 19 symptoms”. A search was carried out on Youtube™ with an extension (Return YouTube dislike) in September 2022 using terms separately, and the 60 first videos with each keyword were analyzed. Repeated videos produced in other languages, news, advertisements, songs, or with content unrelated to the topic, as well as those with more than one hour in length, were excluded [²].

Data Collection

The videos were individually evaluated by two previously trained researchers (Kappa=0.85) who recorded, in an electronic spreadsheet, the following variables: number of likes, number of dislikes, number of comments, length (seconds), engagement [(No. likes – No. dislikes) / total of views x 100%)], rate of view [(No. of views /No. of days since the post of video) X 100], video source (official institutions, health professionals and others), audiovisual quality (low, average, high), reliability, usefulness (CSS) and general quality [⁵].

Evaluation of Reliability, Usefulness (CSS), General Quality and Audiovisual Quality

The reliability of the videos was evaluated using the modified DISCERN scale [⁶]. Reliability concerns the ability of an instrument to consistently display a result or how well an instrument remains free of standard error [¹²]. The scale has five items that assess aspects such as clarity of objectives, use of reliable sources of information, impartiality, presentation of additional sources of information, and evidence of areas of uncertainty on the subject. Each item has a "yes" (one point) or "no" (zero points) response option. The total score for each video can range from 0 to 5, where higher scores indicate greater reliability.

The usefulness of the videos was evaluated through the specific COVID-19 score (CSS), a scale created to assess the usefulness of videos about COVID-19 to the user. The usefulness of the video is related to how useful and informative it will be for the user. On this scale, one point is given for each accurate information about transmission, typical symptoms, prevention strategies, potential treatments, and epidemiology of COVID-19, totaling a maximum of five points. Additionally, each video was defined as true or false. For the video to be considered false, it was enough to have information that did not match the guidelines released by public health agencies [¹⁰].

The overall quality of the video was assessed using the Global Quality Score (GQS), which assigns points ranging from 1 to 5 (GQS: 1=poor quality/not at all useful for the patient; 2=generally poor quality and flow poor/limited use for patients; 3=moderate quality and suboptimal flow/not very useful for patients; 4=good quality and flow/useful for patients; 5=excellent quality/very useful for patients). This score is related to the usefulness of the video for the patient, the way information is transmitted, and the flow (continuity) of the video [¹³,¹⁴].

Audiovisual quality was used to classify videos as good, moderate, or bad. Videos that included visuals, text, professional graphics, clear effects, and noise-free audio, allowing everything to be heard without difficulty, were rated as "good." Home videos were classified as “moderate," with interference from the environment - such as traffic noises - and speeches difficult to understand. Videos with blurred or grainy images that were difficult to understand or had no audio were rated as "bad" [¹⁵].

Statistical Analysis

Data were analyzed using descriptive and inferential statistics. The descriptive analysis comprised the determination of absolute and relative frequencies and measures of central tendency and dispersion. The Mann-Whitney test was used to compare the variables, number of views, likes, dislikes, comments, and engagement between videos with low and moderate/good overall quality. The Mann-Whitney test was used to compare the number of comments, duration of videos, and DISCERN score between videos created or not by health professionals. The Spearman test was used to test the correlation between the quantitative variables collected. All analyses were performed using the Statistical Package for the Social Sciences software (SPSS for Windows, version 25.0, IBM Corp., Armonk, NY, USA) with a significance level of 5%.

Ethical Aspects

The submission to an Ethics Committee in Research with Human Beings was optional, according to Item III, Sole Paragraph, of Article 1 of Resolution 510 of 2016 of the Brazilian National Health Council, as the research concerns public domain information.

Results

Of the 180 videos evaluated, 121 were excluded (42 news, 59 in other languages, 12 repeated, 3 with unrelated content, three songs, one by duration [exceeded 1 hour in duration], one without sound or image), and 59 met the eligibility criteria.

The average quality and reliability scores were 3.0 (±1.1) and 3.2 (±0.8), respectively. Approximately two-thirds of the videos (64.4%) were of low/moderate quality. Most of the videos were about signs and symptoms and biosafety measures. The average likes and views were 23,857 (±61,292) and 518,536 (±886,978), respectively. Regarding the CSS scale, only 5% of the videos had false information (Table 1).

Table 1
Sample characteristics.

Variable	N	%
Global Quality Score (GQS)
1	6	10.2
2	15	25.4
3	17	28.8
4	17	28.8
5	15	6.8
Audiovisual Quality
Good	52	88.1
Moderate	7	11.9
Information Accuracy
True	56	94.9
False	3	5.1
Video Source
Official institutions	1	1.7
Others	20	33.9
Health professionals	38	64.4
Content
Signs and symptoms	29	49.2
Prevention and biosafety	5	8.5
Guidance on seeking care	1	1.7
Signs and symptoms + prevention	10	16.9
Others	14	23.7
	Average (SD)	Median (Min.-Max.)
Views	747758,6 (1085390,2)	335730 (2532-5684207)
Number of likes	30396,2 (71426,4)	6700 (0-504000)
Number of dislikes	595 (1508)	134 (0-10733)
Length	361.6 (299)	266 (30-1224)
Engagement	3.4 (2.45)	2.54 (0-12.4)
DISCERN	3.2 (0.84)	3 (2-5)
CSS	1.5 (0.92)	1 (0-3)

• SD: Standard Deviation; Min.: Minimum; Max.: Maximum.

The number of views, likes, and engagement was significantly higher for videos with moderate/good quality compared to videos with low quality (Table 2). The number of comments, the duration of the videos, as well as the total DISCERN score were significantly higher in videos produced by health professionals (p<0.05) (Table 3).

Table 2
Association between the video's general quality and the number of views, likes, dislikes, comments, and engagement.

General Quality	Views Median (Q25-Q75)	p-value*	Likes Median (Q25-Q75)	p-value*	Dislikes Median (Q25-Q75)	p-value*	Comments Median (Q25-Q75)	p-value*	Engagement Median (Q25-Q75)	p-value*
Low	113133 (12477-341875)	0.005	2000 (279-8400)	0.006	56 (17-266)	0.44	127 (10-869)	0.16	2.16 (1.51-2.49)	0.014
Moderated/Good	490980.5 (159892-1349539)		12000 (3753-41000)		167 (54-832)		1348 (394-2828)		2.995 (2.11-5.05)

• *Mann-Whitney test; Results are significant at the 5% level; Q₂₅ = 25th quartile; Q₇₅ = 75th quartile.

Table 3
Association between video source per healthcare professional and number of views, likes, dislikes, comments, and engagement.

Health Professionals	Comments Median (Q25-Q75)	p-value	Video Length Median (Q25-Q75)	p-value	DISCERN Median (Q25-Q75)	p-value
No	84 (2-690.5)	0.007	168.5 (96.5-282.5)	0.004	2 (2-3)	<0.001
Yes	1327 (400-2345.5)		333 (166-646.5)		3 (3-4)

• *Mann-Whitney test; Results are significant at the 5% level; Q₂₅ = 25th quartile; Q₇₅ = 75th quartile.

There was a statistically significant and positive correlation between DISCERN scores and the number of views, likes, dislikes, comments, and video duration (p<0.05). There was a positive correlation between CSS scores, overall quality, and video duration (p<0.05). Regarding the overall quality of the video, this variable was positively correlated with the number of views, likes, and engagement (p<0.05) (Table 4).

Table 4
Correlation matrix between DISCERN scores, GQS, CSS, number of views, likes, dislikes, comments, length, and engagement of the videos.

	DISCERN	CSS	GQS	Views	Likes	Dislikes	Comments	Length	Engagement
DISCERN	1.000
CSS	0.152	1.000
GQS	0.109	0.300*	1.000
Views	0.350**	-0.91	0.285*	1.000
Likes	0.376**	0.17	0.291	0.932**	1.000
Dislikes	0.271*	-0.84	0.161	0.896**	0.923**	1.000
Comments	0.474**	-0.005	0.246	0.695**	0.768**	0.672**	1.000
Length	0.420**	0.310*	0.171	0.60	0.213	0.118	0.348**	1.000
Engagement	0.250	0.236	0.344**	0.175	0.442**	0.259*	0.551**	0.511**	1.000

• *p<0.05;
• **p<0.01; Statistically significant results are marked in bold; Spearman correlation test.

Discussion

This study investigated the reliability and quality of YouTube videos to determine whether they are useful to users. This is the first study that aims to understand the quality and reliability of Portuguese-language videos on YouTube on the subject, and its strong point is that it uses three validated scales, one specific to COVID-19.

Most of the videos concerned signs and symptoms and biosafety, followed by treatment, diagnosis, and information about seeking medical care. It is worth mentioning that most of the videos addressed more than one theme. Since COVID-19 was an unknown disease, there was a need for a broader approach to understanding the disease. Another factor observed due to the need for more knowledge was the variation of information over time. In older videos, the use of a mask was recommended only for infected people, while in more recent videos, the mask was recommended to everyone.

Engagement is essential for a video to reach the largest number of individuals. The greater the video engagement, the more favorable its position in the platform's search results. Engagement is directly related to the audience's interaction with the video [¹⁶]. In our study, engagement was higher in videos with moderate/good quality when compared to those with low quality. Furthermore, higher scores on the overall quality scale were positively correlated with more views, likes, and engagement. These findings show that video quality is essential to obtain a good reach.

The videos had moderate scores for quality (GQS) and reliability (DISCERN) and were mostly published by health professionals. It is known that YouTube is a powerful resource for health education. However, anyone can post videos on this platform without evaluating their veracity [²,⁵]. Given its high power to spread information, a video with content without scientific support can be dangerous and, consequently, prevent the dissemination of accurate data related to the pandemic [¹⁰]. Fake news impacts people with little resistance to this type of information since they need to verify the source of the publication. This type of information has a 70% greater potential to go viral when compared to accurate news [⁹]. Thus, this video platform should be better used by public agencies and health professionals to produce more valuable and reliable content for the population.

Another finding of this study was that videos published by health professionals had significantly higher reliability than those not made by health professionals. The literature has shown that videos produced by professionals demonstrate greater accuracy, usability, and quality [¹⁰], which may also explain the significant number of comments and the greater engagement observed. The association also indicates that health professionals can explain scientific knowledge in a language appropriate for the general public. In addition, the greater reliability and support of the material produced by health professionals, in association with the fact that the content is about a new disease with significant repercussions in the media, may explain the greater number of likes, comments, and views observed.

Regarding the specific scale of COVID-19, the videos had an average low score, and most of them conveyed accurate information. These findings demonstrate that despite a few videos with false information, there is still a need for more informative material for the public. Another interesting finding was the correlation between the CSS scores and the videos' general quality, which reinforces the validity of this new instrument. Therefore, it is suggested that content creators base themselves on this scale when preparing the videos, responding to each item that makes up the instrument, thus resulting in more complete and valuable videos for society.

In contrast to what was verified in a previous study, which found that more than 25% of the videos in English about COVID-19 presented misleading information [¹⁰], our study observed a small portion of videos with false information (5%). The difference observed between the investigations may be because of the change in the policy of incorrect medical information related to COVID-19 that YouTube adopted on 20 May 2020. The platform does not allow the submission of content that disseminates false medical information that contravenes the guidelines of the World Health Organization, and videos that violate this rule are deleted [¹⁶]. An example of this policy was observed in our study when the platform deleted a video recommending using ivermectin to treat COVID-19.

A limitation of this study is that, as it is a dynamic platform, the YouTube algorithm is based on geographic location, which may result in a variation in the results of video searches [¹⁷]. However, we tried to minimize this problem by using the browser in private mode and turning off the device's location in the search. Another disadvantage is that the study was carried out in a specific period, making its precision concerning time low. In this sense, as the pandemic progresses, additional investigations are necessary to monitor the evolution of the quality, reliability, and usability of the content available to the public. On the other hand, this study's solid methodology, calibration, and use of validated scales are strengths.

Conclusion

Most of the videos showed moderate quality and reliability and a low utility for the population. It is essential to encourage health professionals to produce content, thus generating quality, reliable materials, since YouTube™ can be an important educational resource for propagating useful information to the population.

• Financial Support
  None.

Data Availability

The data used to support the findings of this study can be made available upon request to the corresponding author.

References

• ¹
  Chu JT, Wang MP, Shen C, Viswanath K, Lam TH, Chan SSC. How, when and why people seek health information online: Qualitative study in Hong Kong. Interact J Med Res 2017; 6(2):e24. https://doi.org/10.2196/ijmr.7000
  » https://doi.org/10.2196/ijmr.7000
• ²
  Kilinç DD, Sayar G. Assessment of reliability of youtube videos on orthodontics. Turk J Orthod 2019; 32(3):145-150. https://doi.org/10.5152/TurkJOrthod.2019.18064
  » https://doi.org/10.5152/TurkJOrthod.2019.18064
• ³
  Bora K, Das D, Barman B, Borah P. Are internet videos useful sources of information during global public health emergencies? A case study of YouTube videos during the 2015–16 Zika virus pandemic. Pathog Glob Health 2018; 112(6):320-328. https://doi.org/10.1080/20477724.2018.1507784
  » https://doi.org/10.1080/20477724.2018.1507784
• ⁴
  Andika R, Kao CT, Williams C, Lee YJ, Al-Battah H, Alweis R. YouTube as a source of information on the COVID-19 pandemic. J Community Hosp Intern Med Perspect 2021; 11(1):39-41. https://doi.org/10.1080/20009666.2020.1837412
  » https://doi.org/10.1080/20009666.2020.1837412
• ⁵
  Hutchison CM, Cave V, Walshaw EG, Burns B, Park C. YouTube™ as a source for patient education about the management of dental avulsion injuries. Dent Traumatol 2020; 36(2):207-211. https://doi.org/10.1111/edt.12517
  » https://doi.org/10.1111/edt.12517
• ⁶
  Pons-Fuster E, Roca JR, Tvarijonaviciute A, López-Jornet P. YouTube information about diabetes and oral healthcare. Odontology 2020; 108(1):84-90. https://doi.org/10.1007/s10266-019-00445-3
  » https://doi.org/10.1007/s10266-019-00445-3
• ⁷
  World Health Organization (WHO). WHO Coronavirus (COVID-19) Dashboard. Available from: https://covid19.who.int/ [Accessed on February 10, 2023].
  » https://covid19.who.int/
• ⁸
  Mattos FF, Pordeus IA. COVID-19: a new turning point for dental practice. Braz Oral Res 2020; 34:e085. https://doi.org/10.1590/1807-3107bor-2020.vol34.0085
  » https://doi.org/10.1590/1807-3107bor-2020.vol34.0085
• ⁹
  Galhardi CP, Freire NP, Fagundes MCM, Minayo MCS, Cunha ICKO. Fake news and vaccine hesitancy in the COVID-19 pandemic in Brazil. Cien Saude Colet 2022; 27(05):1849-1858. https://doi.org/10.1590/1413-81232022275.24092021
  » https://doi.org/10.1590/1413-81232022275.24092021
• ¹⁰
  Li HO-Y, Bailey A, Huynh D, Chan J. YouTube as a source of information on COVID-19: A pandemic of misinformation?. BMJ Global Health 2020; 5(5):e002604. https://doi.org/10.1136/bmjgh-2020-002604
  » https://doi.org/10.1136/bmjgh-2020-002604
• ¹¹
  Yuksel B, Cakmak K. Healthcare information on YouTube: Pregnancy and COVID-19. Int J Gynaecol Obstet 2020; 150(2):189-193. https://doi.org/10.1002/ijgo.13246
  » https://doi.org/10.1002/ijgo.13246
• ¹²
  Souza AC, Alexandre NMC, Guirardello EB. Psychometric properties in instruments evaluation of reliability and validity. Epidemiol Serv Saude 2017; 26(3):649-659. https://doi.org/10.5123/S1679-49742017000300022
  » https://doi.org/10.5123/S1679-49742017000300022
• ¹³
  Ozdede M, Peker I. Analysis of dentistry YouTube videos related to COVID-19. Braz Dent J 2020; 31(4):392-398. https://doi.org/10.1590/0103-6440202003767
  » https://doi.org/10.1590/0103-6440202003767
• ¹⁴
  Gul M, Diri MA. YouTube as a source of information about premature ejaculation treatment. J Sex Med 2019; 16(11):1734-1740. https://doi.org/10.1016/j.jsxm.2019.08.008
  » https://doi.org/10.1016/j.jsxm.2019.08.008
• ¹⁵
  Korkmaz YN, Buyuk SK. YouTube as a patient – information source for cleft lip and palate. Cleft Palate Craniofac J 2020; 57(3):327-332. https://doi.org/10.1177/1055665619866349
  » https://doi.org/10.1177/1055665619866349
• ¹⁶
  Google. YouTube Help. Available from: https://support.google.com/youtube?authuser=1#topic=9257498 [Accessed on March 10, 2022].
  » https://support.google.com/youtube?authuser=1#topic=9257498
• ¹⁷
  Refaey K, Tripathi S, Yoon JW, Justice J, Kerezoudis P, Parney IF, et al. The reliability of YouTube videos in patients education for glioblastoma treatment. J Clin Neurosci 2018; 55:1-4. https://doi.org/10.1016/j.jocn.2018.07.001
  » https://doi.org/10.1016/j.jocn.2018.07.001

Edited by

Academic Editor: Wilton Wilney Nascimento Padilha

Publication Dates

• Publication in this collection
  18 Dec 2023
• Date of issue
  2024

History

• Received
  27 Nov 2022
• Accepted
  20 Apr 2023