Academia Research UX Research Chatbot Experience

How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings

Duration

My Role

Responsibility

Sept. 2020 - Sept. 2021

Academia Research

UX Researcher

Qualitative analysis and generate insight

Objective

The popularity of task-oriented chatbots is constantly growing, but smooth conversational progress with them remains profoundly challenging. In recent years, researchers have argued that chatbot systems should include guidance for users on how to converse with them. Nevertheless, empirical evidence about what to place in such guidance, and when to deliver it, has been lacking.

Read Paper➝
‍

Outcome

- It establishes that each guidance type and timing has particular strengths and weaknesses, thus that each type/timing combination has a unique impact on performance metrics, learning outcomes, and user experience.
- On that basis, it presents guidance-design recommendations for future task-oriented chatbots.

Deliverables

- As one of a UX Researcher in this team, I cooperate with 5 UX researchers, and I in charge with qualitative part. We use affinity diagram to analysis the qualitative data and generate insight integrated with quantitative data.
‍

Research Process

01 ➝ 02

➝ 03

➝ 04

➝ 05

Understand Problems

Define the problem and goal

Conducting the research to testing the hypothesis

Analysis the qualitative and quantitative data

Generate insight and design implication for the system

_Backgrounds
Qustion: How to guide task-oriented chatbot users, and when?

Task-oriented chatbots are designed to help users to perform domain-specific tasks. When interacting with this kind of chatbots, efficiency and effectiveness are 2 crucial motivations. Making efficient conversations with chatbots is still a big challenge. It is very often that chatbot might misunderstand the user or just being uncertain about user’s intentions. Such challenge can lead to low trust & satisfaction. Therefore, it’s important for chatbots to guide users through difficulties and engage them in smoother conversations.
‍
In this study, we explored 8 combinations of 2 guidance types and 4 timings, to investigate how chatbot guidance type and timing influence user experience, task performance, and learning outcomes.

Research Questions

RQ1. Which combination of guidance type and timing enables users to
          a)complete their tasks more efficiently,
          b)make better conversational progress, and
          c)improve their performance during subsequent chatbot use?
RQ2. What are users’ subjective experiences of each of these combinations?
RQ3. What are users’ desired characteristics for the combination of a chatbot-conversation guidance type and its timing?

Method

- Using a mixed-methods approach that integrates results from a between-subjects experiment and a reflection session
- This case study compares the effectiveness of eight combinations of two guidance types (example-based and rule-based) at four guidance timings (service-onboarding, task-intro, after-failure, and upon-request), as measured by users’ task performance, improvement on subsequent tasks, and subjective experience.
- 126 Participants (14 participantsx 9 conditions)

Some Results _{(Please Refer to our paper for more details)}

Takeaways

     - Examples warranted a good start, whereas rules promoted understanding.
     - The guidance timing matters.
     - The choices of both guidance type and timing depends on the chatbot’s application and the purpose of the guidance.
‍
‍

Examples of Design Implication _{(Please Refer to our paper for more details)}

- Detect whether the user would be efficiency-oriented or exploration-oriented at any given time-point.
- To accelerate task execution, such chatbots could even proactively shift to example-based guidance when they detect that the user is on the go.
‍
‍