Gauging customer satisfaction with chatbot interactions 💬
Gauging customer satisfaction with chatbot interactions 💬
Evaluative Usability Study
Evaluative Usability Study
OVERVIEW
OVERVIEW
This study aimed to evaluate customer satisfaction when using Amazon Pharmacy's chatbot feature for inquiries about medicine availability, pricing, delivery times, and logistics.
This study aimed to evaluate customer satisfaction when using Amazon Pharmacy's chatbot feature for inquiries about medicine availability, pricing, delivery times, and logistics.
GOALS
GOALS
Assessing Customer Ease of Use
Measuring Quality of Chatbot Responses
Identifying User Pain Points
Making Recommendations
Assessing Customer Ease of Use
Measuring Quality of Chatbot Responses
Identifying User Pain Points
Making Recommendations
The primary objective was to understand how users navigate Amazon Pharmacy's chatbot for common inquiries and
assess the chatbot's effectiveness in providing satisfactory answers to ultimately improve customer experiences.
The primary objective was to understand how users navigate Amazon Pharmacy's chatbot
for common inquiries and
assess the chatbot's effectiveness in providing satisfactory answers to ultimately improve customer experiences.
4 Talk-through interviews
5 tasks to complete
4 Talk-through interviews
5 tasks to complete
Participants aged 18-65.
Novice, current, or previous users of Amazon Pharmacy or similar platforms.
Not exclusive to any age, gender, race, or socioeconomic background.
Pre-screen through a survey to meet criteria
Participants aged 18-65.
Novice, current, or previous users of Amazon Pharmacy or similar platforms.
Not exclusive to any age, gender, race, or socioeconomic background.
Pre-screen through a survey to meet criteria
INTERVIEWS TASKS
INTERVIEWS TASKS
Based on our initial research, we decided that a task-based interview would best yield the behavioral data we needed while also offering valuable insight into the user's thought processes. Consequently, we developed five scenarios to guide the usability study, which allowed us to better understand the diverse needs and expectations of our users.
Based on our initial research, we decided that a task-based interview would best yield the behavioral data we needed while also offering valuable insight into the user's thought processes. Consequently, we developed five scenarios to guide the usability study, which allowed us to better understand the diverse needs and expectations of our users.
1
1
Place an order for medication using the chatbot.
Place an order for medication using the chatbot.
Purpose: Gauge the extent of chatbot usage versus the search bar and assess its effectiveness.
Purpose: Gauge extent of chatbot usage vs the search bar & assess its effectiveness.
2
2
Ask the chatbot for details about medication.
Ask the chatbot for details about medication.
Purpose: Compare chatbot responses to those of a real pharmacist.
Purpose: Compare chatbot responses to those of a real pharmacist.
3
3
Verify medication availability and earliest delivery.
Verify medication availability and earliest delivery.
Purpose: Assess the chatbot's ability to manage stock and delivery information.
Purpose: Assess the chatbot's ability to manage stock and delivery information.
4
4
Inquire about assisting a relative’s medical needs.
Inquire about assisting a relative’s medical needs.
Purpose: Explore broader use cases and gather diverse insights.
Purpose: Explore broader use cases and gather diverse insights.
5
5
Free exploration of the chatbot.
Free exploration of the chatbot.
Purpose: Measure the chatbot’s ability to handle unstructured, varied questions.
Purpose: Measure the chatbot’s ability to handle unstructured, varied questions.
DATA + ANALYSIS
DATA + ANALYSIS
Data Collection
Data was gathered via Zoom, with real-time screen observation and recordings. Quotes and actions were noted to capture user feelings and task completion times.
Data Collection
Data was gathered via Zoom, with real-time screen observation and recordings. Quotes and actions were noted to capture user feelings and task completion times.
Post-Task Interviews
Participants rated their experiences from 1 to 5, providing qualitative data on their satisfaction with the chatbot.
Post-Task Interviews
Participants rated their experiences from 1 to 5, providing qualitative data on their satisfaction with the chatbot.
The data referenced in this report is not private or sensitive. It was collected and analyzed independently, without any association or relation to Amazon or Amazon Pharmacy.
The data referenced in this report is not private or sensitive. It was collected and analyzed independently, without any association or relation to Amazon or Amazon Pharmacy.
QUALITATIVE ANALYSIS
QUALITATIVE ANALYSIS
We went over our interview transcripts and coded our findings into 6 major themes.
We went over our interview transcripts and coded our findings into 6 major themes.
Poor Chatbot Comprehension
Poor Chatbot Comprehension
Poor Chatbot Comprehension
Broad Responses
Broad Responses
Broad Responses
Unclear Instructions for OTC Medication Purchases
Unclear Instructions for OTC Medication Purchases
Unclear Instructions for OTC Medication Purchases
Responses Redirect to
External Links
Responses Redirect to
External Links
Responses Redirect to
External Links
Trouble Answering Medical Questions
Trouble Answering Medical Questions
Trouble Answering Medical Questions
Technical Difficulties
Technical Difficulties
Technical Difficulties
QUANTITATIVE ANALYSIS
QUANTITATIVE ANALYSIS
We also averaged the data from our participant's post-task interviews to gain insight into the overall usability scores for each task.
We also averaged the data from our participant's post-task interviews to gain insight into the overall usability scores for each task.
Chatbot's Helpfulness
Chatbot's Helpfulness
"On a scale of 1-5, with 1 being not helpful at all and 5 being very helpful, to what extent was the chatbot helpful in assisting you through the process?"
"On a scale of 1-5, with 1 being not helpful at all and 5 being very helpful, to what extent was the chatbot helpful in assisting you through the process?"
Task 1
Ordering a medication & adding it to cart.
Task 1
Ordering a medication & adding it to cart.
Helpfulness: 1.25
Helpfulness: 1.25
Overall Experience
Overall Experience
"On a scale of 1-5, 1 being very hard and 5 being very easy, please rate your experience of getting information from the chatbot. Why did you choose this rating?"
"On a scale of 1-5, 1 being very hard and 5 being very easy, please rate your experience of getting information from the chatbot. Why did you choose this rating?"
Tasks 2
Inquire about safe dosage, risks, etc…
Tasks 2
Inquire about safe dosage, risks, etc…
Overall Experience: 1
Overall Experience: 1
Tasks 3
Verifying the availability of medicine & earliest procurement dates.
Tasks 3
Verifying the availability of medicine & earliest procurement dates.
Overall Experience: 1.25
Overall Experience: 1.25
Tasks 4
Inquire about assisting with a relative’s medical needs.
Tasks 4
Inquire about assisting with a relative’s medical needs.
Overall Experience: 2.25
Overall Experience: 2.25
Quality of Information
Quality of Information
"On a scale of 1-5, 1 being “the information was unhelpful or irrelevant to your question” to 5 being “the information was great, comprehensive, and exactly what you asked for”, please rate the quality of information from the chatbot. Why did you choose that rating?"
"On a scale of 1-5, 1 being “the information was unhelpful or irrelevant to your question” to 5 being “the information was great, comprehensive, and exactly what you asked for”, please rate the quality of information from the chatbot. Why did you choose that rating?"
Tasks 2
Inquire about safe dosage, risks, etc…
Tasks 2
Inquire about safe dosage, risks, etc…
Quality of Info: 1
Quality of Info: 1
Tasks 3
Verifying the availability of medicine & earliest procurement dates.
Tasks 3
Verifying the availability of medicine & earliest procurement dates.
Quality of Info: 1.5
Quality of Info: 1.5
Tasks 4
Inquire about assisting with a relative’s medical needs.
Tasks 4
Inquire about assisting with a relative’s medical needs.
Quality of Info: 2.25
Quality of Info: 2.25
User Satisfaction with Chatbot Use
User Satisfaction with Chatbot Use
"On a scale of 1-5, 1 being not satisfied at all, and 5 being very satisfied, how would you rate your satisfaction with the chatbot’s ability to answer your question? "
"On a scale of 1-5, 1 being not satisfied at all, and 5 being very satisfied, how would you rate your satisfaction with the chatbot’s ability to answer your question? "
Task 5
Free Exploration
Task 5
Free Exploration
Satisfaction: 1.25
Satisfaction: 1.25
RECOMMENDATIONS
RECOMMENDATIONS
To enhance user satisfaction when interacting with Amazon Pharmacy's Chatbot feature, we developed four key recommendations.
To enhance user satisfaction when interacting with Amazon Pharmacy's Chatbot feature, we developed four key recommendations.
Concise Answers
Concise Answers
Improve the chatbot to provide specific answers rather than redirecting to lengthy articles. This could be implemented with the help of a generative AI technology.
Pain Points Addressed
Improve the chatbot to provide specific answers rather than redirecting to lengthy articles. This could be implemented with the help of a generative AI technology.
Pain Points Addressed
Broad Responses
Broad Responses
Responses Redirect to
External Links
Responses Redirect to
External Links
New Redesign
Integrated natural language processing (NLP) tools and a GPT AI model
New Redesign
Integrated natural language processing (NLP) tools and a GPT AI model
Current Interface
Keyword-matching response generation linking to external help articles
Current Interface
Keyword-matching response generation linking to external help articles
Rephrased Question Options
Rephrased Question Options
Provide examples and clickable buttons for common queries to help users phrase questions more effectively, allowing the chatbot to respond accurately.
Pain Points Addressed
Provide examples and clickable buttons for common queries to help users phrase questions more effectively, allowing the chatbot to respond accurately.
Pain Points Addressed
Poor Chatbot Comprehension
Poor Chatbot Comprehension
Broad Responses
Broad Responses
New Redesign
Prompt user with options to rephrase their question for clarity.
New Redesign
Prompt user with options to rephrase their question for clarity.
Current Interface
Chatbot provides excessive information due to unclear understanding of questions.
Current Interface
Chatbot provides excessive information due to unclear understanding of questions.
Enhanced Medical Information
Enhanced Medical Information
Develop an up-to-date database for the chatbot to provide basic information about the most common drugs, enhancing the relevance and precision of its responses.
Pain Points Addressed
Develop an up-to-date database for the chatbot to provide basic information about the most common drugs, enhancing the relevance and precision of its responses.
Pain Points Addressed
Unclear Instructions for OTC Medication Purchases
Unclear Instructions for OTC Medication Purchases
Trouble Answering Medical Questions
Trouble Answering Medical Questions
Potential Tool for Implimentation
2800+ medical concepts + 110,000 hours of clinical validation
Potential Tool for Implimentation
2800+ medical concepts + 110,000 hours of clinical validation
⚠️ Disclaimer
Relying only on AI assistants for medical advice, including OTC drugs, risks inaccurate diagnoses and inappropriate treatments, as they lack the personalized assessment of healthcare professionals.
⚠️ Disclaimer
Relying only on AI assistants for medical advice, including OTC drugs, risks inaccurate diagnoses and inappropriate treatments, as they lack the personalized assessment of healthcare professionals.
Address Technical Issues
Address Technical Issues
Fix technical problems, including chat limits and query comprehension, to improve the overall user experience of interacting with the chatbot.
Fix technical problems, including chat limits and query comprehension, to improve the overall user experience of interacting with the chatbot.
Pain Points Addressed
Technical Difficulties
Technical Difficulties
