Evaluating insomnia queries from an artificial intelligence chatbot for patient education.

Publication/Presentation Date

4-1-2024

Abstract

STUDY OBJECTIVES: We evaluated the accuracy of ChatGPT in addressing insomnia-related queries for patient education and assessed ChatGPT's ability to provide varied responses based on differing prompting scenarios.

METHODS: Four identical sets of 20 insomnia-related queries were posed to ChatGPT. Each set differed by the context in which ChatGPT was prompted: no prompt, patient-centered, physician-centered, and with references and statistics. Responses were reviewed by 2 academic sleep surgeons, 1 academic sleep medicine physician, and 2 sleep medicine fellows across 4 domains: clinical accuracy, prompt adherence, referencing, and statistical precision, using a binary grading system. Flesch-Kincaid grade-level scores were calculated to estimate the grade level of the responses, with statistical differences between prompts analyzed via analysis of variance and Tukey's test. Interrater reliability was calculated using Fleiss's kappa.

RESULTS: The study revealed significant variations in the Flesch-Kincaid grade-level scores across 4 prompts: unprompted (13.2 ± 2.2), patient-centered (8.1 ± 1.9), physician-centered (15.4 ± 2.8), and with references and statistics (17.3 ± 2.3,

CONCLUSIONS: ChatGPT can be used to generate clinically accurate responses to insomnia-related inquiries.

CITATION: Alapati R, Campbell D, Molin N, et al. Evaluating insomnia queries from an artificial intelligence chatbot for patient education.

Volume

20

Issue

4

First Page

583

Last Page

594

ISSN

1550-9397

Disciplines

Business Administration, Management, and Operations | Health and Medical Administration | Management Sciences and Quantitative Methods

PubMedID

38217478

Department(s)

Administration and Leadership

Document Type

Article

Link to Full Text

Share

COinS