EVALUATION OF THE PERFORMANCE OF CHATGPT/ARTIFICIAL INTELLIGENCE IN THE MULTIPLE-CHOICE TEST TO OBTAIN THE TITLE OF SPECIALIST IN ORTHOPEDICS AND TRAUMATOLOGY

AVALIAÇÃO DO DESEMPENHO DO CHATGPT/INTELIGÊNCIA ARTIFICIAL NA PROVA DE MÚLTIPLAS ESCOLHAS PARA OBTENÇÃO DO TÍTULO DE ESPECIALISTA EM ORTOPEDIA E TRAUMATOLOGIA

ABSTRACT

Introduction:  ChatGPT, an advanced Artificial Intelligence model specialized in natural language processing, shows remarkable abilities, achieving high scores in certification exams in various specialties. This study aims to evaluate ChatGPT’s performance in multiple-choice tests applied to obtain specialist certification in Orthopedics and Traumatology.

Methods:  We used ChatGPT 4.0 to answer 100 questions from the first phase of the Título de Especialista em Ortopedia e Traumatologia 2022 (TEOT) (Specialist in Orthopedics and Traumatology Test). We excluded non-text-based questions. Each question was entered individually into ChatGPT, with a new session initiated for each question. Performance was evaluated regarding number of words and questions’ taxonomic classification.

Results:  Of the 95 questions analyzed, ChatGPT answered 61.05% correctly and 38.95% incorrectly. There was no statistically significant difference regarding number of words, and ChatGPT’s performance did not vary according to taxonomic level.

Conclusion:  ChatGPT demonstrated vast knowledge in Orthopedics, with acceptable performance in the TEOT exam. Results suggest ChatGPT’s an educational and clinical resource in Orthopedics, but needs future progress and human supervision for its effective application. Level of evidence IV, Case series.

Keywords:
Artificial Intelligence; Orthopedics; Medical Education