WED-069 - Evaluating the Acceptability and Usability of AI Chatbots in Providing Health Information to Young Adults
Wednesday, April 16, 2025
6:00 PM – 7:00 PM PST
Location: Pacific I/II, 2nd Floor
Area of Responsibility: Area VI: Communication Subcompetencies: 6.4 Select methods and technologies used to deliver message(s)., 6.6 Evaluate communication. Research or Practice: Research
Student Western Michigan University Kalamazoo, Michigan, United States
Learning Objectives:
At the end of this session, participants will be able to:
Identify at least two specific generative AI chatbots and evaluate their effectiveness for delivering real-time health information to young adults by the end of the presentation.
Analyze the readability, trustworthiness, and user-friendliness of AI chatbot health information responses using at least two evaluation criteria discussed during the presentation.
Describe how user feedback on AI chatbots' compassion and trustworthiness can inform improvements in health communication strategies by the conclusion of the presentation.
Brief Abstract Summary: Learn how popular generative AI chatbots like ChatGPT, Google Gemini, Claude, and Perplexity are evaluated for delivering real-time, evidence-based health information to young adults aged 15-35. This study explores participant feedback on chatbot responses to a common health question. Acceptability is measured by trustworthiness, compassion, and likelihood of future use, while usability focuses on grammar and readability. The findings highlight key factors influencing chatbot effectiveness, such as conversational tone and simplicity, and can guide improvements in chatbot design and future research, aiming to enhance health literacy and access to care through AI technology.
Detailed abstract description:
Introduction: As generative AI technology continues to evolve, its potential to provide accessible, evidence-based health information in real-time has become a growing area of interest. Chatbots like ChatGPT, Google Gemini, Claude, and Perplexity offer a novel way for individuals, particularly those aged 15-35, to seek immediate answers to health-related queries. For this population, who may prefer instant digital resources over in-person healthcare interactions, these tools could fill an important gap. This study aims to explore the acceptability and usability of AI chatbots in delivering health information by analyzing user feedback and measuring chatbot performance across various dimensions.
Methods: This survey-based study involved collecting responses from four popular generative AI chatbots (ChatGPT, Google Gemini, Claude, and Perplexity) to a frequently searched health question relevant to the 15-35 year age group. An estimated 30 participants will be recruited to review the AI-generated responses. Participants will be provided transcripts of each chatbot's response. Acceptability will be measured by perceived trustworthiness, compassion, likelihood of future use, and usability will be measured by analyzing the grammar and readability using standard tools to assess the ease of comprehension.
Results: Data will be analyzed using descriptive statistics to summarize participant feedback. Ratings of chatbot trustworthiness, ease of use, and readability will be compared to determine which AI system appears most effective for young adults seeking health information. Preliminary findings will be examined to see if preferences vary for different chatbot features, such as conversational tone and simplicity. Readability metrics will be used to further evaluate the accessibility of the health information provided.
Conclusion: The findings of this study will help determine the utility of AI chatbots in providing real-time health information to young adults. Based on the results, recommendations for improving chatbot interfaces and functionalities can be made. Future research will involve a larger sample size to better understand how AI can be integrated into public health strategies to enhance health literacy and access to care.
