Challenges associated with the use of ChatGPT in health researchDear Editor,
Vieira et al.(1) report titled "ChatGPT: immutable insertion in health research and researchers’ lives", and Daungsupawong et al.(2) subsequent comments on that report consider the use of artificial intelligence (AI)—specifically ChatGPT—in health research, and discuss the advantages that this technology offers. Daungsupawong et al.(2) complement Vieira et al.(1) report by suggesting the association of new technological resources to expand the application of AI in health research. Both articles warn of potential errors and failures associated with the use of these tools that may arise in the future. However, some important considerations that have been broached should be addressed in greater depth, such as the risk of "hallucinations" and the generation of incorrect responses by AI.(3)
The term "hallucination" is used to describe situations in which AI platforms such as ChatGPT create information that was not in the training data, and does not align with reality. The phenomenon is particularly concerning in areas such as healthcare, where the accuracy and reliability of information are crucial. If not properly detected and corrected, these hallucinations can compromise the quality of research, lead to errors in diagnosis and treatment, and hinder decision-making by healthcare professionals.(4)
It is essential to acknowledge that AI does not have the capacity to discern the truthfulness of the sources it uses to generate its responses, thus it may incorporate outdated, inaccurate, or false data. This underscores the importance of researchers maintaining a critical view when using these tools, in an effort to ensure that the information generated is rigorously verified before it is applied in scientific or clinical contexts.(5)
In conclusion, while AI offers significant potential to transform healthcare research, it is essential that researchers, professionals, and the broader academic community are aware of the present limitations of its use.
-
AUTHORS’ STATEMENT ON GENERATIVE ARTIFICIAL INTELLIGENCE
The authors declare that the AI tool "ChatGPT" was used during the generation of this report, but assert that it was solely used to assist in preliminary translation, linguistic revision, and formatting of the text. All intellectual content, data interpretation, and scientific conclusions were entirely developed by the authors.
DATA AVAILABILITY
The content is already available.
REFERENCES
- 1 Vieira AG, Saconato H, Eid RA, Nawa RK. ChatGPT: immutable insertion in health research and researchers’ lives [letter]. einstein (São Paulo). 2024;22(Spec 1):eCE0752.
- 2 Daungsupawong H, Wiwanitkit V. ChatGPT and the future of medical education: Correspondence [letter]. J Taibah Univ Med Sci. 2023;19(2):248-9.
- 3 Hatem R, Simmons B, Thornton JE. A Call to Address AI "Hallucinations" and How Healthcare Professionals Can Mitigate Their Risks. Cureus. 2023;15(9):e44720.
-
4 Mendel UMass Amherst. AI Hallucinations Impact Medical Summaries. Clinical Trials Arena. 2024 [cited 2024 Nov 29]. Available from: https://www.clinicaltrialsarena.com
» https://www.clinicaltrialsarena.com -
5 AR5IV. Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI. ar5iv.org. 2024 [cited 2024 Nov 29]. Available from: https://ar5iv.org
» https://ar5iv.org
-
Associate Editor:
Kenneth J. Gollob Hospital Israelita Albert Einstein, São Paulo, SP, Brazil ORCID: https://orcid.org/0000-0003-4184-3867
