The Performance of ChatGPT on the American Society for Surgery of the Hand Self-Assessment Examination.

Publication/Presentation Date

4-1-2024

Abstract

BACKGROUND: This study aims to compare the performance of ChatGPT-3.5 (GPT-3.5) and ChatGPT-4 (GPT-4) on the American Society for Surgery of the Hand (ASSH) Self-Assessment Examination (SAE) to determine their potential as educational tools.

METHODS: This study assessed the proportion of correct answers to text-based questions on the 2021 and 2022 ASSH SAE between untrained ChatGPT versions. Secondary analyses assessed the performance of ChatGPT based on question difficulty and question category. The outcomes of ChatGPT were compared with the performance of actual examinees on the ASSH SAE.

RESULTS: A total of 238 questions were included in the analysis. Compared with GPT-3.5, GPT-4 provided significantly more correct answers overall (58.0% versus 68.9%, respectively

CONCLUSIONS: GPT-4 demonstrated improved performance over GPT-3.5 on the ASSH SAE, especially on more difficult questions. Actual examinees scored higher than both versions of ChatGPT, but the margin was cut in half by GPT-4.

Volume

16

Issue

4

First Page

58950

Last Page

58950

ISSN

2168-8184

Disciplines

Medicine and Health Sciences

PubMedID

38800302

Department(s)

Department of Surgery

Document Type

Article

Link to Full Text

Share

COinS