Evaluation of the feasibility of artificial intelligence models as patient information advisors for human papillomavirus vaccines

Kaya, Abdulkadir; Kücükdag, Hüseyin Nejat; Keyif, Betül

doi:10.1590/1806-9282.20250809

home Sumário navigate_before Anterior Atual Seguinte navigate_next

home Sumário

Original Article • Rev. Assoc. Med. Bras. 71 (11) • 2025 • https://doi.org/10.1590/1806-9282.20250809 linkcopy

Evaluation of the feasibility of artificial intelligence models as patient information advisors for human papillomavirus vaccines

Authorship SCIMAGO INSTITUTIONS RANKINGS

SUMMARY

OBJECTIVE: Artificial intelligence chatbots are increasingly used to disseminate health information. The aim of this study was to evaluate the accuracy, reliability, quality, and readability of responses generated by four artificial intelligence chatbots regarding human papillomavirus vaccination.

METHODS: Frequently asked questions about human papillomavirus vaccination were identified using a Google search tool, and these questions were posed to the ChatGPT-3.5, Gemini, Copilot artificial intelligence, and ChatGPT-4 models. Responses were assessed for accuracy (five-point Likert scale), reliability (modified DISCERN scale), quality (Global Quality Scale), and readability (Flesch Reading Ease Score). Interobserver agreement was evaluated by the intraclass correlation coefficient. The results were evaluated at a significance level of p<0.05. SPSS 23.0 package programme was used in the analyses.

RESULTS: There were significant differences between chatbots in terms of accuracy (p=0.001), reliability (p<0.001), and quality (p<0.001), but no significant difference in readability (p=0.497). ChatGPT-4 demonstrated the highest accuracy and quality, while Copilot artificial intelligence demonstrated superior reliability. All models produced responses that were moderately difficult to read. The intraclass correlation coefficient values for inter-rater reliability ranged from 0.034 to 0.512.

CONCLUSION: Artificial intelligence chatbots show promising potential for use as patient information counsellors regarding human papillomavirus vaccination. However, improvements in readability and consistent evidence-based content generation are required before widespread clinical application.

KEYWORDS:
Artificial intelligence; HPV vaccine; Patient education; Chatbots; Health communication

		Chatbots				p*
		C1	C2	C3	C4	p*
Accuracy	Mean±SD	4.66±0.52	4.33±0.70	4.49±0.62	4.68±0.52	0.001
Accuracy	Median (IQR)	5 (1)	4 (1)	5 (1)	5 (1)	0.001
Reliability	Mean±SD	4.31±0.51	4.01±0.66	4.42±0.62	4.33±0.56	<0.001
Reliability	Median (IQR)	4 (1)	4 (0)	4 (1)	4 (1)	<0.001
Quality	Mean±SD	4.28±0.60	3.96±0.67	3.99±0.59	4.63±0.53	<0.001
Quality	Median (IQR)	4 (1)	4 (0)	4 (0)	5 (1)	<0.001
Readability	Mean±SD	82.94±8.96	82.50±9.28	80.72±9.90	81.06±9.07	0.497
Readability	Median (IQR)	80 (10)	85 (10)	80 (20)	80 (20)	0.497

Chatbots	Chatbots	Accuracy	Reliability	Quality
C1	C2	0.004	0.005	0.005
	C3	0.281	0.719	0.008
	C4	1.000	1.000	<0.001
C2	C1	0.004	0.005	0.005
	C3	0.529	<0.001	1.000
	C4	0.001	0.003	<0.001
C3	C4	0.156	0.895	<0.001

Criteria	Chatbots	Intraclass correlation coefficient 95%CI	p
Accuracy	C1	0.391 (0.113–0.612)	0.004
	C2	0.424 (0.159–0.634)	0.001
	C3	0.500 (0.244–0.691)	<0.001
	C4	0.374 (0.093–0.600)	0.005
Reliability	C1	0.110 (-0.110 to 0.346)	0.165
	C2	0.471 (0.211–0.669)	<0.001
	C3	0.423 (0.156–0.634)	0.002
	C4	0.131 (-0.082 to 0.364)	0.051
Quality	C1	0.246 (-0.034 to 0.494)	0.043
	C2	0.512 (0.265–0.697)	<0.001
	C3	0.347 (0.075–0.574)	0.007
	C4	0.163 (-0.137 to 0.435)	0.142
Readability	C1	0.235 (-0.029 to 0.479)	0.034
	C2	0.301 (-0.030 to 0.565)	0.002
	C3	0.034 (-0.071 to 0.180)	0.276
	C4	0.275 (-0.098 to 0.612)	<0.001

vertical_align_top show_chart

more_horiz close
- image
- translate
- link
- article
- vertical_align_top
- file_download
- show_chart
- image
- translate
- link
- article

location_on

Associação Médica Brasileira R. São Carlos do Pinhal, 324, 01333-903 São Paulo SP - Brazil, Tel: +55 11 3178-6800, Fax: +55 11 3178-6816 - São Paulo - SP - Brazil
E-mail: ramb@amb.org.br

rss_feed Acompanhe os números deste periódico no seu leitor de RSS

[TFN1] C1: ChatGPT-3.5; C2: Gemini; C3: Copilot AI; C4: ChatGPT-4; SD: standard deviation; IQR: interquartile range.

[TFN2] ^*
Kruskal-Wallis analysis. The significant values are presented in bold.