Assessment of websites on cleft lip and palate in Brazilian Portuguese

ABSTRACT

Purpose:

this study aimed at assessing the legibility, reliability, usability and coverage of websites on cleft lip and/or palate (CL/P) in Brazilian Portuguese and providing a global comparison.

Methods:

in order to evaluate the obtained data, four protocols were used, i.e., the Website Coverage Protocol, the legibility (by means of the COH METRIX PORT), the reliability (Discern Protocol), and the usability (by means of the System Usability Scale).

Results:

98 websites were included with average legibility scoring of 31.6±11.7, rated as a hard readability level. Coverage average scoring was 19.9±3.79 from a total of 35 points. As for reliability, average scoring was 43.2±6.51 points, rated as acceptable. The usability scoring was 24.8±2.18 points, rated as the worst imaginable one.

Conclusion:

therefore, the assessed Brazilian websites on CL/P were rated difficult for legibility, requiring high-school level of education, restricted coverage and acceptable reliability, in addition to the low usability. The results were similar to those of websites from other countries, except for the legibility aspect, which was lower for Brazilian websites.

Keywords:
Cleft Lip; Cleft Palate; Patient Portals; Health Care Quality, Access, and Evaluation

INTRODUCTION

Expectations and great changes arise for parents from pregnancy. This process of baby idealization can be disrupted when parents face a malformation diagnosis, in addition to doubts on the child development ¹. Pointed as the most frequent congenital anomaly, the cleft lip and/or palate (CL/P), is a craniofacial malformation, multifactorial etiology comprising the lip, alveolar ridge, hard and soft palate, occurring, worldwide, in one to each 700 born alive babies every year². It is related to abnormalities in major structures of the oral cavity, hearing disorders, feeding problems, apart from the risk for the speech and language development and consequences for physical appearance^1,3,4.

With the diagnosis, during the pre-natal period, the family has to deal with the “bereavement” of the idealized baby, but also starts the search for information¹. When receiving the medical diagnosis, the Internet is the research source for the patient who is after problem-solving, people who underwent the same situation, and even other procedures from those recommended by the patient’s doctor or healthcare professional. A study shows that 25% of the netizens search for websites and videos on health conditions, and 80% search for information on treatment of specific diseases⁵. This behavior is important to the extent that it empowers the population for decision-making, as well as to exercise their right of choice on health matters. However, such rights are hindered by their difficulty in selecting the most updated, scientific evidence-based information sources on the Internet⁶.

The integration of high-complexity and primary healthcare services is fundamental for investments in health-promotion actions, and telehealth and tele-education are interesting ways for such actions to cover the greatest possible number of people, not only among the population at large⁷, but also among health professionals⁸.

Although the Internet enables more knowledge spread, facilitating the access to information by patients and professionals, there is still a great number of websites that provide data in a complex language⁹, difficult accessibility and large content variety on CL/P¹⁰.

Websites are analyzed in different ways and the attempt to standardize this process has been increasingly frequent in literature discussions, so that the population makes use of these materials judiciously. Readability is an important characteristic that indicates how easy a document is to read. This aspect may limit the audience that would have access to this material, in cases where there is a lower instructional level^11,12. The quality of the content, considering everything from ethical aspects¹³ to the coverage of the topics covered, added to the ease of use of the layout of each website¹⁴, make up relevant points to guarantee different characteristics that would facilitate the information to reach the user effectively.

Therefore, this study aimed to assess the legibility, reliability, usability and coverage of websites on CL/P in Brazilian Portuguese and provide a global comparison.

METHODS

This research study comprised the production of documental data, available and free access on the Internet, not being necessary to submit it to the Ethics Board.

The search was held by three evaluators by means of the following search engines: GoogleTM (www.google.com), BingTM (www.bing.com) e YAHOO!® (www.yahoo.com), in October 2021 and updated in October 2022, using the search keywords: 1-“palate fissure”, 2-“lip fissure”, 3-“lip and palate fissure", 4-“cleft lip”, 5-“cleft palate”, 6-“cleft lip and palate”, 7-“hare lip” and 8-"wolf’s jaw”. Websites in Portuguese were included with available and accessible language for users, shown up to the 5th page of each database, providing information on CL/P, excluding scientific, academic websites, personal opinion sites, discussion groups and forums, in addition to videos on the subject.

About the evaluators, all were females, 21 years old, in the last year of speech-language pathology graduation. These evaluators had the task of selecting the websites (at the Google search, Bing and Yahoo search engine) and evaluating them blindly, through the Website Coverage Protocol, Discern Protocol and the System Usability Scale. In cases of conflict, consensus meetings were held together with the two experts supervising the work, speech-language pathologists, ten years graduated, with clinical experience in cleft lip and palate.

Each evaluator searched in one of the mentioned databases, individually; 143 websites were initially found in the Google search engine, 161 in the Bing search engine, and 176 in the Yahoo search engine, totaling 481. After the removal of duplicate websites, and the application of the exclusion criteria, the comparison and analysis of the obtained data were performed, resulting in the final assessment of 98 websites. The websites were arranged in a list in ascending numerical order according to their position in each search engine (order the websites were located), for the organization of the exhibition of the results of this research.

In order to assess the legibility, the COH METRIX PORT tool was used, adapted from the Coh-Metrix tool for Brazilian Portuguese¹¹. The Coh-Metrix tool calculates the indexes in order to assess text cohesion, coherence and comprehension difficulty, using several levels of linguistic analysis: lexical, syntactic, discursive and conceptual. This tool is automatic from Microsoft Windows, so it was not necessary for the evaluators to carry it out in a blinded way, being generated in just one check. To do this, a text in TXT format is inserted and a series of counts are returned, such as FLESH INDEX (an intelligibility index). Its indexes range from 0 to 100, being 0 to 25 rated as a very difficult reading level, requiring higher-education level of schooling, matching specific academic fields; from 25 to 50, difficult reading level, that is, from high-school to higher education level of schooling; from 50 to 75, an easy level, that is, compatible from the 5th to the 9th grades; and from 75 to 100, a very easy reading level, understandable for those with level of schooling from the 1st to the 4th grades of elementary school¹².

The Website Coverage Protocol was developed by the authors of this research, to analyze seven aspects by means of a scale from 1 to 5 (score 1 means the worst performance, and score 5, the best performance): definition, causes and consequences, classification, guidance to parents on development, feeding and treatment for the CL/P provided by a speech-language pathologist. First, the experts met with the evaluators to calibrate the essential topics in each of the topics that would be evaluated on the coverage of the websites. After clarifying the topics, the evaluators accessed all sections of each website, following the graduation proposed for this aspect.

The same care was taken with the Discern Protocol and System Usability Scale. Before the evaluators started analyzing and scoring the websites, a meeting was held with the experts to study the questionnaires and clarify the parameters to be used. For this training, the application of the protocols was exemplified in a site that had been excluded, to resolve doubts and standardize this judgment.

The Discern Protocol is used in research in the health area, exclusively. With it, it is possible to create an analysis and content assessment pattern of a publication. The protocol comprises 16 questions divided in three groups. Questions 1 to 8 assess the text reliability; questions 9 to 15 assess the text quality, and question 16 is the general assessment of the information. Each question is responded in a scale from 1 to 5, in which the lowest number means that the assessed aspect is not present in the text, while the highest number means that the assessed aspect is fully present in the text. By adding up each question, it is possible to rate the websites in 5 categories: very poor (scores between 16 and 26), poor (between 27 and 38), fair (between 39 and 50), good (between 51 and 62), and excellent (over 63)¹³.

The System Usability Scale (SUS) is a questionnaire with 10 questions used to assess the effectiveness, efficiency and satisfaction of users towards websites. The odd statements indicate the positive points in this category, and the even statements show usability problems. Scoring may vary from 0 to 100, and rating can be attributed as the worst imaginable from 0 to 25, poor from 25 to 39, OK from 39 to 52, good from 52 to 74, excellent from 74 to 85, and the best imaginable from 85 to 100¹⁴.

In order to compare the data obtained with the world panorama, a literature review was carried out in the PubMed/Medline database with the following search strategy:

("evaluability"[All Fields] OR "evaluate"[All Fields] OR "evaluated"[All Fields] OR "evaluates"[All Fields] OR "evaluating"[All Fields] OR "evaluation"[All Fields] OR "evaluation s"[All Fields] OR "evaluations"[All Fields] OR "evaluative"[All Fields] OR "evaluatively"[All Fields] OR "evaluatives"[All Fields] OR "evaluator"[All Fields] OR "evaluator s"[All Fields] OR "evaluators"[All Fields] OR ("evaluability"[All Fields] OR "evaluate"[All Fields] OR "evaluated"[All Fields] OR "evaluates"[All Fields] OR "evaluating"[All Fields] OR "evaluation"[All Fields] OR "evaluation s"[All Fields] OR "evaluations"[All Fields] OR "evaluative"[All Fields] OR "evaluatively"[All Fields] OR "evaluatives"[All Fields] OR "evaluator"[All Fields] OR "evaluator s"[All Fields] OR "evaluators"[All Fields])) AND ("website"[All Fields] OR "website s"[All Fields] OR "websites"[All Fields] OR "site"[All Fields]) AND ("cleft lip and palate"[All Fields] OR "cleft palate"[All Fields] OR "cleft lip"[All Fields])

Studies that had the objective of evaluating websites about cleft lip and palate and that had used at least one of the four parameters adopted in this study (readability, coverage, reliability and usability) were included, not being mandatory to use the same instruments that were used.

The questionnaire results were tabulated and described in terms of percentage, median and standard deviation. Website assessment studies from other countries were searched in the literature and their results were compared to the ones in the current study. Statistical analysis was conducted by means of Anova Test in order to compare Discern ratings to the other parameters, using the Post-hot Tukey´s Test. Pearson´s correlation was used for the correlation between all evaluations. The significance level adopted was 5%. Jamovi, version 1.2.25, was the selected program.

RESULTS

Ninety-eight (98) websites on child CL/P were analyzed in Brazilian Portuguese. The term used which found the greatest number of websites was “cleft lip and/or palate”. Chart 1 shows the number of websites found by keywords and search engines.

Chart 1
Selection of websites by keywords used in three search engines, applying the inclusion criteria and removing duplicates

Keywords	Search engines
	Google	Bing	Yahoo!
Palate fissure	20	20	23
Lip fissure	20	22	26
Lip and palate fissure	12	23	24
Cleft palate	23	21	22
Cleft lip	17	21	21
Cleft lip and palate	24	27	26
Hare lip	21	24	28
Wolf’s jaw	7	3	6
Total	481 websites
Included	98 websites

The 98 analyzed websites were listed below.

W1 - https://www.arevistadamulher.com.br/faq/28872-labio-leporino-entenda-as-causas-e-conheca-os-tratamentos

W2 - https://www.asaudeempauta.com/2015/09/que-e-labio-leporino-fissura-labial-goela-de-lobo-fenda-palatina-labiopalatal.html

W3 - https://www.abc.med.br/

W4 - http://abflp.org.br/fissuras-labio-palatinas/

W5 - http://afissore.org.br/fissura-labiopalatina

W6 - https://artedecuidar.webnode.com.br/products/sobre%20as%20fissuras%20labio-palatais%20-%20classifica%C3%A7%C3%A3o/

W7 - https://www.ativosaude.com/saude/labio-leporino-fenda-palatina/

W8 - https://www.atualfm.com.br/site/fenda-palatina-e-labio-leporino-vamos-falar-sobre/

W9 - https://brasil.babycenter.com/a2900031/l%C3%A1bio-leporino-e-fenda-palatina

W10 - https://bebemamae.com/gravidez/saude-gravidez/labio-leporino-o-que-e-causas-tratamento-e-mais

W11 - https://bebe.abril.com.br/saude/tudo-sobre-labio-leporino-malformacao-congenita-que-e-tratavel/

W12 - http://bvsms.saude.gov.br/dicas-em-saude/2071-fissura-labio-palatal-e-labio-leporino

W13 - https://leiturinha.com.br/blog/bebe-com-labio-leporino/

W14 - http://www.blog.saude.gov.br/index.php/34887-saude-da-crianca-labio-leporino

W15 - https://blog.segurosunimed.com.br/fissura-labial-fenda-palatina/

W16 - https://drauziovarella.uol.com.br/doencas-e-sintomas/labio-leporino-fissura-labial-e-fenda-palatina/

W17 - https://margaridagp.blogspot.com/2014/10/fendas-labiais-e-palatinas.html

W18 - https://brasilescola.uol.com.br/saude/labios-leporinos.htm

W19 - https://cabecadecrianca.ig.com.br/conheca-cinco-mitos-sobre-a-fissura-labiopalatina/

W20 - https://catracalivre.com.br/saude-bem-estar/cinco-mitos-sobre-labio-leporino-e-fenda-palatina/

W21 - http://centroexcelencia.com.br/tag/fenda-labiopalatina/

W22 - https://cientificalab.com.br/saude/fissura-labial-e-fenda-palatina

W23 - http://www.clariceabreu.com.br/atuacao/cirurgia-craniomaxilofacial/cirurgia-craniofacial-pediatrica/fissura-de-labio-e-palato-labio-leporino-e-fenda-palatina/

W24 - https://www.colgate.com.br/

W25 - https://www.crechesegura.com.br/crianca-com-fissura-labial-e-fenda-palatina-na-escola/

W26 - https://criancaespecial.com.br/por-que-alguns-bebes-nascem-com-labio-leporino-eou-fenda-palatina/

W27 - https://criancagenial.blogspot.com/2008/04/lbio-leporino-ou-goela-de-lobo.html#gsc.tab=0

W28 - https://www.culturamix.com/saude/doencas/labio-leporino/

W29 - https://www.naturalcura.com.br/labio-leporino/

W30 - https://www.cursosaprendiz.com.br/labio-leporino/

W31 - http://www.dapheneozelame.com.br/fissura-labiopalatina/

W32 - http://www.derepentegravida.com.br/labio-leporino-o-que-e-o-que-causa-como-e-cirurgia-fotos/

W33 - https://www.saudedica.com.br/labio-leporino-o-que-e-causas-e-tratamento/

W34 - https://www.educarsaude.com/

W35 - https://www.educare.pt/opiniao/artigo/ver/?id=157244

W36 - https://www.enfermagemnovidade.com.br/2017/11/labio-leporino-e-fenda-palatina.html

W37 - https://exenin.com/cara-boca-e-garganta/fenda-labial-e-fissura-palatina-ou-fissuras-orofaciais/

W38 - https://fissuradanamaternidade.com/2014/10/30/afinal-o-que-e-a-fissura-labiopalatina/

W39 - https://fofuuu.com/

W40 - https://folhabiologica.bio.br/arquivos/1033

W41 - http://ctmc.lusiada.br/malformacoes-craniofaciais/

W42 - http://www.fundef.org.br/pagina.php?cont=especialidadesFissura

W43 - https://www.gestacaobebe.com.br/fenda-palatina-causas-e-tratamento

W44 - https://gouodonto.com.br/blog/ortodontia/labio-leporino-as-maiores-mentiras-e-verdades/

W45 - https://www.gndi.com.br/o-que-e-fissura-labiopalatina

W46 - https://www.guiadobebe.com.br/labio-leporino/

W47 - https://br.guiainfantil.com/labio-leporino.html

W48 - https://guiamedicobrasileiro.com.br/labio-leporino-e-fenda-palatina/

W49 - http://hrac.usp.br

W50 - http://www.hnscpm.org.br/blog/motivo-de-preocupacao-de-pais-labio-leporino-e-fenda-palatina-podem-ser-tratados-e-nao-provocar-sequelas

W51 - https://www.hospitalsaocamilosp.org.br/sua-saude-agradece/labio-leporino-como-a-correcao-cirurgica-e-realizada

W52 - https://hospitalsiriolibanes.org.br/imprensa/noticias/Paginas/Lebio-leporino-ou-palato-fendido-o-que-e.aspx

W53 - https://sobrapar.org.br/tratamentos/fissuras/fissuras-labio-palatinas/

W54 - https://www.huggies.com.br/gravidez/a-ecografia-no-diagnostico-do-labio-leporino

W55 - http://www.ipjc.com.br/fissura-labiopalatal/

W56 - https://www.infoescola.com

W57 - http://www.itad.pt/fenda-labio-palatina/

W58 - https://institutopensi.org.br/blog-saude-infantil/labio-leporino-e-fenda-palatina/

W59 - https://www.jornaldosite.com.br/arquivo/anteriores/ivan/artivan74.htm

W60 - https://www.kenhub.com/pt/library/anatomia/desenvolvimento-do-palato

W61 - https://www.liviascelza.com/fissura-labiopalatina

W62 - https://www.localodonto.com.br/mitos-e-verdades-sobre-fissura-labial/

W63 - https://maesemulheres.com.br/fissura-labial-e-fissura-palatina-qual-a-diferenca/

W64 - https://www.msdmanuals.com/pt/profissional/pediatria/anormalidades-craniofaciais-e-musculoesquel%C3%A9ticas-cong%C3%AAnitas/l%C3%A1bio-leporino-e-fenda-palatina

W65 - https://medicoresponde.com.br/fenda-palatina-quais-as-causas-e-como-tratar/

W66 - http://www.meumenino.com.br/fissura-labial/

W67 - https://www.minhavida.com.br

W68 - https://minutosaudavel.com.br/fenda-palatina/

W69 - https://gravidez.online/labio-leporino-tratamento-causas/

W70 - https://mundoeducacao.uol.com.br/biologia/labio-leporino.htm

W71 - https://www.news-medical.net/health/Breastfeeding-and-Cleft-Palate-Cleft-Lip-(Portuguese).aspx

W72 - https://www.odontoblogia.com.br/labio-leporino/

W73 - https://opas.org.br/fenda-palatina-labio-leporino-o-que-e-causas-e-tratamento/

W74 - https://www.prematuridade.com/index.php/noticia-mod-interna/fissura-labio-palatina-operacao-sorriso-brasil-6501

W75 - http://prolactare.com/amamentacao/amamentacao-e-fisssura-labiopalatina

W76 - http://www.saudedescomplicada.com/clinica-geral/goela-de-lobo-quais-seus-tratamentos-e-sintomas-e-perigoso-confira/

W77 - https://www.saudedica.com.br/labio-leporino-o-que-e-causas-e-tratamento/

W78 - https://www.uai.com.br/app/noticia/saude/colunistas/alexandre-meira/2020/03/16/noticias-saude,256701/labio-leporino-em-criancas-e-uma-situacao-comum-e-tem-tratamento.shtml

W79 - https://blog.segurosunimed.com.br/fissura-labial-fenda-palatina/

W80 - https://www.smiletrainbrasil.com/pt-br/stories/o-que-%C3%A9-fissura-labiopalatina

W81 - http://www2.cirurgiaplastica.org.br/cirurgias-e-procedimentos/labio-leporino/

W82 - https://www.sorrisologia.com.br

W83 - https://soumamae.com.br/labio-leporino-o-que-e-consequencias/

W84 - https://souenfermagem.com.br/termos-tecnicos/letra-f/fenda-palatina/

W85 - https://www.terra.com.br/vida-e-estilo/saude/saude-bucal/atualidades/entenda-o-que-e-fissura-labial-e-fenda-palatina,baa4be4870211410VgnVCM10000098cceb0aRCRD.html

W86 - https://www.trocandofraldas.com.br

W87 - https://www.tuasaude.com/labio-leporino-e-fenda-palatina/

W88 - https://www.ricardoshimosakai.com.br/labio-leporino-podem-causar-deficiencias-respiratorias-de-fala-e-audicao/

W89 - https://depto.icb.ufmg.br/dmor/Disciplinas/Embriologia/labio.htm

W90 - https://www.vix.com

W91 - https://www.wavemedicinafetal.com.br/fissura-labiopalatina/

W92 - https://pt.wikipedia.org/wiki/Fenda_labial_e_palatina

W93 - https://saudebucal.yahoo.com/o-que-e-fissura-labial-palatina-210034018.html

W94 - https://asfissuradas.com/

W95 - http://portaldosbebes.fob.usp.br/portaldosbebes/Portugues/detSubCategoriaInstitucional.php?codsubcategoria_fono=10047&codcategoria_site=1

W96 - https://fissuraeaudicao.wordpress.com/

W97 - http://camiladininno.com.br/

W98 - http://fissuralabiopalatina.unb.br/

Regarding the legibility, the assessed websites obtained the mean score of 31.6, rated as difficult reading level, requiring high school to higher education levels of schooling from users. The minimum achieved was 6.51, and the maximum was 57.1.

In the information coverage on CL/P, definition, cause, consequences, classification, guidance for parents and treatment can be highlighted. Definition was the highest rated one, while CL/P treatment had the lowest rating, as described in Table 1. Some websites also inform the medical specialties which treat CL/P: pediatricians, surgeons, oral and maxillofacial surgeons and plastic surgeons, otorhinolaryngologists, dentists, speech-language pathologists, psychologists, nutritionists, physical therapists, nurses, social workers and educators. Other websites also mention geneticists, obstetricians, traumatologists as specialists who treat CL/P.

Table 1
Information coverage of websites on clef lip and palate

Coverage	Median±SD	Minimum	Maximum
Definition	3.31±0.76	1.5	5
Cause	3.27±0.92	1.0	5
Outcomes	3.09±0.81	1.33	4.66
Classification	2.48±0.96	1.0	5
Guidance for parents on development	2.77±0.80	1.33	4.66
Guidance for parents on feeding	2.61±1.08	1.0	5
Treatment, including speech-language pathology, for CL/P	2.34±0.83	1.0	4.66
Total	19.0±3.78	13.0	32.3

• Captions: SD - Standard Deviation; CL/P - cleft lip and/or palate.

On average, the usability of the websites was rated “the worst imaginable” (Table 2), with scores ranging from 19.32 to 29.67 (maximum scoring)¹⁴.

Table 2
Usability of the website information on cleft lip and/or palate ¹⁴

Usability (SUS)	Mean±SD	Minimum	Maximum
Q.1	2.91±0.91	1	5
Q.2	1.30±0.34	1	2.33
Q.3	4.44±0.37	3.5	5
Q.4	1.21±0.32	1	2.33
Q.5	3.40±0.83	1	5
Q.6	1.31±0.29	1	2.33
Q.7	4.43±0.50	1	5
Q.8	1.35±0.31	1	2.33
Q.9	3.39±0.80	1.33	5
Q.10	1.0±0.12	1	1.66
Total	24.8±2.19	19.3	29.67

• Captions: SUS - System Usability Scale; Q - question; SD - Standard Deviation.

The websites obtained scores from 28.68 (minimum scores) to 63.34. Reliability by means of the Discern was considered fair on average (Table 3).

Table 3
Information reliability of the websites on cleft lip and/or palate ¹³

DISCERN	Mean SD	Minimum	Maximum
Q1	3.90±0.63	2.33	5
Q2	3.33±0.74	1.66	5
Q3	3.61±0.76	1.33	5
Q4	2.70±1.31	1	5
Q5	3.01±1.53	1	5
Q6	3.68±0.72	1.66	5
Q7	1.93±0.82	1	5
Q8	1.38±0.37	1	3
Q9	2.27±0.78	1	5
Q10	1.00±0.06	1	1.66
Q11	3.02±0.74	1.33	4.66
Q12	2.81±0.65	1.33	5
Q13	2.81±0.65	1.33	5
Q14	1.92±0.69	1	4.5
Q15	2.71±0.73	1	4.66
Q16	2.29±0.82	1	5
Q17	3.33±0.66	1.66	5
Total	43.0±6.62	28.7	63.34

• Captions: Q - question; SD - Standard Deviation.

Table 4 shows that coverage as well as usability had the highest scores for higher ranks of reliability by means of the Discern¹³.

Table 4
Mean values, standard deviation and correlations of the Discern rating¹³, according to rank, legibility, coverage and usability

Rating DISCERN	Rank	Legibility	Coverage	Usability
Very poor a	-	-	-	-
Poor b	12.4±7.72	33.2±8.96	17.6±3.15	23.0±1.71
Fair c	13.3±6.66	31.4±13.0	19.8±2.78	25.0±1.70
Good d	12.7±6.02	28.3±7.31	25.1±2.62	28.2±0.9.
Very good e	11.3±4.93	31.9±8.15	29.3±4.64	28.7±1.20
Anova Test	0.915	0.616	0.001*	< 0.001*
Tukey´s Test	-	-	b;c;d;e	b;c;d;e

• Captions: Anova Test and Post-Hoc Tukey´s Test, in order to compare the four groups in the Discern rating; (*) <0.05 statistical significance.

Website W92 had the highest score in two categories assessed in this study, coverage (32.33) and reliability (63.34). To legibility, the highest score was to W44 with 57.1% and to usability was W12 with 29.67.

These data were compared to website assessments on the studied theme from other four countries, described in Chart 2. Only two studies, from the USA and Switzerland, presented an evaluation of three parameters, the others presented an evaluation of one or two parameters. In general, the other countries presented a similar evaluation to the Brazilian websites.

Chart 2
Comparison of website assessment parameters on cleft lip and/or palate from different countries

Country	Number of websites	Legibility	Coverage	Reliability	Usability
Brazil (this study)	98 websites included	The average scoring was 31.6 - rated as difficult reading level (high school to higher education in order to understand its reading)	The average scoring was 19.0 from a total of 35 points.	The average scoring by means of the DISCERN was 43 points, rated as fair.	The average scoring of the websites was 24, that is, the worst imaginable.
USA (Afonso et al., 2019)	111 specific websites of reference teams on CL/P	10.7 (reading level corresponding to the 10th to the11th grades, which exceeds the recommended sixth grade level by the American Medical Association (AMA)	The average content scoring was 14.5 from a total of 20 points	-	In 47 (28.1%), the links did not work, and 17 (10.2%) did not have accessible website.
USA (Wasserburg et al.,2021)	220 websites	The legibility level did not reach the standard seventh-grade level recommended by the National Institutes of Health (NIH). In the keyword search for “Palate fissure”, the reading level corresponded to high school, and in the keyword search for “Lip fissure”, it matched eighth grade- reading level. Flesch reading ease index was 56.99% on average.	-	-	-
USA (De Felippe and Kar, 2015)	15 websites	Fletch-Kincaid resulted in an average scoring of 8.93 (9th grade-reading level)	-	-	-
United Kingdom (Rangaraju et al., 2021)	46 websites included	-	-	DISCERN scores ranged from 17 to 76 for “Lip fissure”, 25 to 78 for “Cleft palate”, and 17 to 78 for “Lip and palate fissure”.	-
Greece (Karamitros and Kitsos, 2018)	146 websites *included websites from several countries (USA, United Kingdom, Australia, South Africa, New Zealand, India, Israel and Canada)	-	34 websites covered 22 items sufficiently (high scoring).	Mean score of 19 points (36 points - maximum scoring). 34 (23.3%) websites had high scores. 112 (76.7%) had low scores.	-
Switzerland (Antonarakis and Kitsos, 2018)	49 websites *included websites from several countries (92% from the USA)	Mean of reading level corresponding to eight and ninth grades	33% did not mention orthodontics. 67% had different amounts of information on the role of orthodontics for patients with CL/P. 9% with enough amount of information on orthodontics for CL/P; 29% with only one or two sentences stating that orthodontic treatment is usually required; 29% of the websites included brief discussion on orthodontics, without enough information, though.	45% authorship attributed to universities or hospitals; 22% listed references.	-

• Caption: CL/P - cleft lip and/or palate.

DISCUSSION

There is an expressive offer of websites on CL/P in Brazilian Portuguese (Chart 1), although with different qualities in terms of legibility, reliability, usability and coverage. Thus, it is relevant that their contents be certified, with reliable information, fundamentally for a genetic disorder so frequently occurred worldwide as the CL/P.

Regarding legibility, the provision of clear, legible contents is one of the main points for any website design, with good experience of use. Most of the assessed websites evidenced a difficult level for understanding, which may limit the access to information. Thus, not only aesthetic issues should be considered, but also the way the text has been written, can make the reader lose interest in the content, takes longer to read it or even cannot understand the information clearly. Therefore, the legibility level was rated as difficult, requiring schooling compatible to high school or higher education, corroborating an American study ¹⁰, but contradicting two others American studies^9,15 and one Swiss study¹⁶, which evidenced an easier reading level on CL/P. However, the influence of a significantly smaller sample of websites in the two latter studies^15,16 should be considered for comparison purposes with the current study.

Legibility is important to be considered when assessing websites; however, it cannot be an isolated parameter, since it deals with quantitative metrics, syllable and word count¹². Content understanding goes beyond quantitative aspects. Exemplifying, a study in which parents accessed websites on CL/P with different levels of legibility, contrastingly, they could obtain similar understanding of their contents⁵.

By assessing the content coverage of 98 Brazilian websites, it was observed that information on treatment had the lowest score, followed by guidance for parents and classification of CL/P (Table 1), corroborating a study from Switzerland, which pointed content gaps on treatment, specifically on the role of orthodontistry¹⁶.

There was variability in Brazilian websites regarding the professionals involved in the treatment of patients with CL/P. Literature was observed to support that variability of professionals specialized in treating CL/P^16,17. This aspect should be considered during guidance to patients/parents/guardians. Additionally, website refinement is also suggested, aiming at greater coverage on the topic of specialized professionals for patients to be referred to.

The assessment on the usability of the websites had an average scoring of 24.8, rated as “the worst imaginable” (Table 2), which corroborates an American study that found that 28.1% of websites on CL/P did not work¹⁰. Ensuring the usability of Internet sources is important for their contents to be explored by greater number of people from different age ranges and different levels of experience/ease towards the Information and Communication Technologies¹⁰.

As for the reliability (Discern protocol), most websites included in this study were considered fair (Table 3). One of the lowest scored questions was number 8, which asks if the website mentions other information sources, corroborated by the results of a Swiss study, which found that 78% of the websites did not cite other references to access¹⁶. Comparing to a Greek study¹⁸, similar results were also found, with average scoring of 50% from the total scoring of the questionnaire used. However, when compared to websites from the USA ⁶ the maximum scoring of the Brazilian study was lower.

Still about the reliability, the analysis by rank enabled to evidence the influence of different parameters to assess the websites, stressing the importance of a multivariate assessment, and such evaluations agreed with each other (Table 4). The diversity of the adopted analyses in studies from different countries should also be pointed out^{6,9,10,15,16,18} (Chart 2), and agreement on the protocols and variables for the evaluation of the information on the Internet is deemed necessary.

The highest rated website (W92) in two categories is a recurring research source, also ranking first in the search engine (rank page), despite its limitation regarding legibility and usability, important factors for greater accessibility.

The importance of the current study lies in the assessment of the impact of quality information online for laypeople, in an attempt to verify if proper information on CL/P has been provided in order to guide patients and their caregivers. Moreover, the findings in this study may guide further research for the creation of a scientific evidence-based website, with relevant legibility and quality under ethical principles, in order to provide people with detailed information on CL/P. As a limitation of this research, it should be noted that there is still no standardization of the protocols used to evaluate websites in general, at a national and international level, which makes it difficult to strengthen and advance investigations in this regard.

CONCLUSION

The assessed Brazilian websites on CL/P were rated difficult for legibility, requiring high-school level of education, restricted coverage and acceptable reliability, in addition to the low usability. When comparing Brazilian websites with those of other countries, it was found that the aspects of coverage, usability and reliability were similar, while legibility was lower for Brazilian websites.

REFERENCES

• ¹
  Nasreddine G, El Hajj J, Ghassibe-Sabbagh M. Orofacial clefts embryology, classification, epidemiology, and genetics. Mutat Res Rev Mutat Res. 2021;787:108373. https://doi.org/10.1016/j.mrrev.2021.108373 PMID: 34083042.
  » https://doi.org/10.1016/j.mrrev.2021.108373
• ²
  Lewis CW, Jacob LS, Lehmann CU. Section on oral health. The primary care pediatrician and the care of children with cleft lip and/or cleft palate. Pediatrics. 2017;139(5):e20170628. https://doi.org/10.1542/peds.2017-0628 PMID: 28557774.
  » https://doi.org/10.1542/peds.2017-0628
• ³
  Maximino LP, Marcelino FC, Cavalheiro MG, Abramides DVM, Caldana ML, Corrêa CC et al. Auditory and language skills in children with cleft lip and palate. Acta Otorrinolaringol Esp (Engl Ed). 2022;73(3):157-63. https://doi.org/10.1016/j.otoeng.2020.11.005 PMID: 35577431.
  » https://doi.org/10.1016/j.otoeng.2020.11.005
• ⁴
  Madhoun LL, Crerand CE, O'Brien M, Baylis AL. Feeding and growth in infants with cleft lip and/or palate: relationships with maternal distress. Cleft Palate Craniofac J. 2021;58(4):470-8. https://doi.org/10.1177/1055665620956873 PMID: 32924577.
  » https://doi.org/10.1177/1055665620956873
• ⁵
  Jodeh DS, Pringle AJ, Rottgers SA. Comprehension of online educational material regarding orofacial clefts. Cleft Palate Craniofac J. 2019;56(9):1206-12. https://doi.org/10.1177/1055665619839576 PMID: 30943779.
  » https://doi.org/10.1177/1055665619839576
• ⁶
  Rangaraju M, Lloyd MS, Khan K. Empowerment of parents of children with cleft lip and palate by objective scoring of patient information websites. J Craniofac Surg. 2021;32(6):2050-2. https://doi.org/10.1097/SCS.0000000000007631 PMID: 33770038.
  » https://doi.org/10.1097/SCS.0000000000007631
• ⁷
  Corrêa CC, Freire T, Zabeu JS, Martins A, Ferreira R, Franscisconi PAS et al. Teleducation about cleft lip and palate: an interdisciplinary approach in the promotion of health. Int Arch Otorhinolaryngol. 2015;19(2):106-11. https://doi.org/10.1055/s-0034-1544114 PMID: 25992163.
  » https://doi.org/10.1055/s-0034-1544114
• ⁸
  Maximino LP, Zambonato TCdF, Picolini-Pereira MM, Corrêa CdC, Feniman MR, Blasca WQ. Development and evaluation of a blog about cleft lip and cleft palate and hearing. Int Arch Otorhinolaryngol. 2018;22(1):60-7. https://doi.org/10.1055/s-0037-1603494 PMID: 29371900.
  » https://doi.org/10.1055/s-0037-1603494
• ⁹
  Wasserburg JR, Sayegh F, Sanati-Mehrizy P, Graziano FD, Taub PJ. Cleft care readability: can patients access helpful online resources? Cleft Palate Craniofac J. 2021;58(10):1287-93. https://doi.org/10.1177/1055665620980244 PMID: 33325255.
  » https://doi.org/10.1177/1055665620980244
• ¹⁰
  Alfonso AR, DeMitchell-Rodriguez EM, Ramly EP, Noel DY, Levy-Lambert D, Wang MM et al. Assessment of American Cleft Palate-Craniofacial Association-Approved Teams' Websites for patient-oriented content and readability. Cleft Palate Craniofac J. 2019;56(9):1213-9. https://doi.org/10.1177/1055665619850441 PMID: 31129984.
  » https://doi.org/10.1177/1055665619850441
• ¹¹
  Wolfe CR, Dandignac M, Wang C, Lowe SR. Gist Inference Scores Predict Cloze Comprehension "In Your Own Words" for native, not ESL readers. Health Commun. 2022;37(14):1757-64. https://doi.org/10.1080/10410236.2021.1920690 PMID: 33947301.
  » https://doi.org/10.1080/10410236.2021.1920690
• ¹²
  Piña SP, Corrêa CdC, Carvalho LRd, Weber SAT. Comprehensiveness, readability, and reliability of Brazilian websites available for lay people's guidance on adenotonsillectomy. Braz J Otorhinolaryngol. 2021;87(1):66-73. https://doi.org/10.1016/j.bjorl.2019.06.012 PMID: 31439530.
  » https://doi.org/10.1016/j.bjorl.2019.06.012
• ¹³
  Passos KK, Leonel AC, Bonan PR, Castro JF, Pontual ML, Ramos-Perez FM et al. Quality of information about oral cancer in Brazilian Portuguese available on Google, Youtube, and Instagram. Med Oral Patol Oral Cir Bucal. 2020;25(3):e346. https://doi.org/10.4317/medoral.23374 PMID: 32040464.
  » https://doi.org/10.4317/medoral.23374
• ¹⁴
  Liang J, Xian D, Liu X, Fu J, Zhang X, Tang B et al. Usability study of mainstream wearable fitness devices: feature analysis and system usability scale evaluation. JMIR mHealth and uHealth. 2018;6(11):e11066. https://doi.org/10.2196/11066 PMID: 30409767.
  » https://doi.org/10.2196/11066
• ¹⁵
  De Felippe N, Kar F. Readability of information related to the parenting of a child with a cleft. Interact J Med Res. 2015;4(3):e4210. https://doi.org/10.2196/ijmr.4210 PMID: 26155814.
  » https://doi.org/10.2196/ijmr.4210
• ¹⁶
  Antonarakis GS, Kiliaridis S. Internet-derived information on cleft lip and palate for families with affected children. Cleft Palate Craniofac J. 2009;46(1):75-80. https://doi.org/10.1597/07-206.1 PMID: 19115798.
  » https://doi.org/10.1597/07-206.1
• ¹⁷
  Guerrero-Abello P, Ariza-Araujo Y, Caycedo-García DJ, Pachajoa H. The need for clinical guidelines for the comprehensive management of patients with cleft lip and palate. Rev Salud Publica (Bogota). 2016;18(1):82-94. https://doi.org/10.15446/rsap.v18n1.41884 PMID: 28453156.
  » https://doi.org/10.15446/rsap.v18n1.41884
• ¹⁸
  Karamitros GA, Kitsos NA. Clefts of the lip and palate: is the Internet a trustworthy source of information for patients? Int J Oral Maxillofac Surg. 2018;47(9):1114-20. https://doi.org/10.1016/j.ijom.2018.03.003 PMID: 29622479.
  » https://doi.org/10.1016/j.ijom.2018.03.003

• A study conducted at Centro Universitário Planalto do Distrito Federal (UNIPLAN), Brasília, Distrito Federal, Brasil.
• Financial support: Nothing to declare.

Publication Dates

• Publication in this collection
  09 Oct 2023
• Date of issue
  2023

History

• Received
  27 Feb 2023
• Accepted
  01 Sept 2023