Comparison of ChatGPT and Traditional Patient Education Materials for Men's Health.
Publication/Presentation Date
1-1-2024
Abstract
INTRODUCTION: ChatGPT is an artificial intelligence platform available to patients seeking medical advice. Traditionally, urology patients consulted official provider-created materials, particularly the Urology Care Foundation™ (UCF). Today, men increasingly go online due to the rising costs of health care and the stigma surrounding sexual health. Online health information is largely inaccessible to laypersons as it exceeds the recommended American sixth to eighth grade reading level. We conducted a comparative assessment of patient education materials generated by ChatGPT vs UCF regarding men's health conditions.
METHODS: All 6 UCF men's health resources were identified. ChatGPT responses were generated using patient questions obtained from UCF. Adjusted ChatGPT responses were generated by prompting, "Explain it to me like I am in sixth grade." Textual analysis was performed using sentence, word, syllable, and complex word count. Six validated formulae were used for readability analysis. Two physicians independently scored responses for accuracy, comprehensiveness, and understandability. Statistical analysis involved Wilcoxon matched-pairs test.
RESULTS: ChatGPT responses were longer and more complex. Both UCF and ChatGPT failed official readability standards, although ChatGPT performed significantly worse across all 6 topics (all
CONCLUSIONS: When comparing readability, ChatGPT-generated education is less accessible than provider-written content, although neither meets the recommended level. Our analysis indicates that specific artificial intelligence prompts can simplify educational materials to meet national standards and accommodate individual literacy.
Volume
11
Issue
1
First Page
87
Last Page
94
ISSN
2352-0787
Published In/Presented At
Shah, Y. B., Ghosh, A., Hochberg, A. R., Rapoport, E., Lallas, C. D., Shah, M. S., & Cohen, S. D. (2024). Comparison of ChatGPT and Traditional Patient Education Materials for Men's Health. Urology practice, 11(1), 87–94. https://doi.org/10.1097/UPJ.0000000000000490
Disciplines
Business Administration, Management, and Operations | Health and Medical Administration | Management Sciences and Quantitative Methods
PubMedID
37914380
Department(s)
Administration and Leadership
Document Type
Article