Evaluating ChatGPT responses on obstructive sleep apnea for patient education.
Publication/Presentation Date
12-1-2023
Abstract
STUDY OBJECTIVES: We evaluated the quality of ChatGPT responses to questions on obstructive sleep apnea for patient education and assessed how prompting the chatbot influences correctness, estimated grade level, and references of answers.
METHODS: ChatGPT was queried 4 times with 24 identical questions. Queries differed by initial prompting: no prompting, patient-friendly prompting, physician-level prompting, and prompting for statistics/references. Answers were scored on a hierarchical scale: incorrect, partially correct, correct, correct with either statistic or referenced citation ("correct+"), or correct with both a statistic and citation ("perfect"). Flesch-Kincaid grade level and citation publication years were recorded for answers. Proportions of responses at incremental score thresholds were compared by prompt type using chi-squared analysis. The relationship between prompt type and grade level was assessed using analysis of variance.
RESULTS: Across all prompts (n = 96 questions), 69 answers (71.9%) were at least correct. Proportions of responses that were at least partially correct (
CONCLUSIONS: ChatGPT overall provides appropriate answers to most questions on obstructive sleep apnea regardless of prompting. While prompting decreases response grade level, all responses remained above accepted recommendations for presenting medical information to patients. Given ChatGPT's rapid implementation, sleep experts may seek to further scrutinize its medical literacy and utility for patients.
CITATION: Campbell DJ, Estephan LE, Mastrolonardo EV, Amin DR, Huntley CT, Boon MS. Evaluating ChatGPT responses on obstructive sleep apnea for patient education.
Volume
19
Issue
12
First Page
1989
Last Page
1995
ISSN
1550-9397
Published In/Presented At
Campbell, D. J., Estephan, L. E., Mastrolonardo, E. V., Amin, D. R., Huntley, C. T., & Boon, M. S. (2023). Evaluating ChatGPT responses on obstructive sleep apnea for patient education. Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine, 19(12), 1989–1995. https://doi.org/10.5664/jcsm.10728
Disciplines
Business Administration, Management, and Operations | Health and Medical Administration | Management Sciences and Quantitative Methods
PubMedID
37485676
Department(s)
Administration and Leadership
Document Type
Article