Serving customers through chatbots: positive and negative effects on customer experience

2.1 Customer experience (CX)

CX is the customers’ subjective response to any interaction with that harmonise services and goods to engage customers in memorable events and integrate material and emotional consumption (Schmitt et al., 2015). Prior analyses of this construct highlight different dimensions (relational, cognitive, affective and behavioural) related to customers’ deliberate and spontaneous reactions at various touchpoints (McColl-Kennedy et al., 2019). Relational CX pertains to how customers perceive their relationships with other actors (employees, customers, technologies) or entities (brands) during interactions (Gahler et al., 2023). These relationships can range from casual acquaintanceships to strong connections that shape the social contexts of customers’ interactions. Cognitive CX relates to the efficiency of obtaining products and services; it involves functional information (e.g. service quality) that helps customers assess and decide on purchases (Barari et al., 2020). Affective CX encompasses customers' emotional responses – such as excitement, pleasure, relaxation, security and entertainment – during their interactions with service providers. This dimension particularly focuses on the enjoyment and hedonic engagement in shopping experiences, as highlighted by Gao et al. (2023). Whilst affective responses vary widely, customers often classify them into categories of pleasure (like happiness and love) or displeasure (such as anger and sadness), as noted by Gao et al. (2021). On the other hand, behavioural CX pertains to the actions and reactions customers exhibit, reflecting their lifestyles, interactions with others and their broader environment (Rather and Hollebeek, 2021).

CX may be pleasant or unpleasant “with regard to satisfaction, value, quality, image, purchase intentions, patronage, loyalty and recommendations” (Jain et al., 2017 p. 655). Scholars argue that CX reactions correlate to positive consumption outcomes (e.g. satisfaction, repurchase behaviour), whereas negative reactions relate to negative consumption outcomes (e.g. dissatisfaction, avoidance behaviour) (Manthiou et al., 2020). From this standpoint, technology plays a crucial role (Flavián et al., 2019). For example, the ability to access information instantly and from anywhere and to participate in e-commerce has changed the ways that people buy – sometimes without even speaking to service providers (Gao et al., 2021). Emerging technologies, such as AI and chatbots enhance and facilitate the ability of service providers to respond to new types of customer needs, such as immersive, tailored and emotional experiences (Hoyer et al., 2020). Customers experience positive feelings when they rely on technology, which effectively completes required tasks, even when it replaces human operators (Huang and Rust, 2022).

However, customers respond negatively when their cognitive and affective goals are not met (Barari et al., 2020). For example, if customers perceive they are spending more time than necessary using technology, a negative CX results, characterised by negative emotions of frustration, disappointment and uncertainty (McLean et al., 2018). Feelings of stress may also occur when using technology and stressful experiences can reduce customers’ affective responses and satisfaction (Lucia-Palacios et al., 2020). Kumar et al. (2022) describe customers’ “technostress” as “the result of negative experiences faced by customers when interacting with firm-based new-age technologies” (p. 2240). Such negative experiences arise if customers do not know how to use a technology or place no trust in it. The discussion surrounding the impact of smart technologies on CX is intensifying, with the emphasis shifting towards the role of digital service encounters.

2.2 Digital service encounters

Service encounters are rapidly evolving, as the result of vast technological advances, causing a paradigm shift in the typically “low-tech, high-touch” context (Bitner et al., 2000). “Digital service encounters” (Heinonen, 2008) or “Service Encounter 2.0” (Larivière et al., 2017) can pertain to “any customer–company interaction that results from a service system that is composed of interrelated technologies (either company- or customer-owned), human actors (employees and customers), physical/digital environments and company/customer processes” (Larivière et al., 2017, p. 239). Such service encounters generally occur online, encompass interactions with a variety of technologies and are increasingly shaped by the customer’s active role (Maar et al., 2023). De Keyser et al. (2019) define various archetypes of technology-infused service encounters, highlighting the replacement of frontline employees with technology as the most common encounter, particularly as a result of the rapid development of AI-based assistants. Moreover, as Robinson et al. (2020) note, as “AI continues to become more human-like, opportunities for counterfeit service encounters will increase”. In counterfeit service encounters, customers and frontline employees are unaware they are speaking with non-human partners.

In this evolving hybrid phygital context (Mele and Russo Spena, 2021), the quality of the interaction between customers and technology can be critical in enabling satisfactory service encounters. In service research, interaction quality refers to consumers’ perceptions of how service is provided (Castelo et al., 2023) and how consumers view their relationships with service providers, such as staff members, during service delivery (Kim and Choi, 2016). Interaction quality engenders trust as a critical factor that influences the adoption of AI-based voice-assisted systems. Trust relates to consumers’ perceptions that virtual agents perform consistently and thus inspire feelings of confidence when providing support (Fernandes and Oliveira, 2021; Lee et al., 2021).

As the digital reliability of service quality increases, customers switch to technology-based service providers, seeking to avoid the service failures experienced during other (human operator) service encounters (Wu et al., 2021). Not all service encounters result in positive CXs and negative events increase the likelihood of customers exiting the encounters (Akaka et al., 2015). Negative emotions and stress may accumulate if negative situations persist (Fliess and Volkers, 2020). For task-oriented customers, a correlation emerges between shopping stress and journey abandonment as service quality worsens. Service encounters may have a significant impact on CX if customers end up worse off than they were before they used the service (Bolton et al., 2022). For example, service failure might lead to a perceived loss of resources (e.g. wasted time) that results in frustration and mistrust (Castillo et al., 2021). Although such studies shed some light on the adverse outcomes of technologies, no research has focused on chatbots and CX, although such understanding could be useful given the growing deployment of these conversational agents (Van Pinxteren et al., 2020; Söderlund and Oikarinen, 2021).

2.3 Chatbots in service research

Emerging field research has focused on smart services that affect CX through connected systems and machine intelligence (Beverungen et al., 2019). Conversational agents represent a specific type of smart service because they enable “real-time data collection, continuous communication and interactive feedback” (Wünderlich et al., 2015, p. 443). They incorporate computational-linguistics methods into online communication settings, as “text-based virtual robots that emulate human-to-human conversation through natural language processing” (Mozafari et al., 2021, p. 221). The expression ‘conversational agent” implies the exchange of ideas, viewpoints, or emotions that typically takes place during conversations that use agent-designated systems to simulate human interaction. However, since the debut of these agents, they have been assigned a variety of labels that influence how they interact with clients, including virtual assistant, digital assistant, conversational agent and chatbot (e.g. Van Pinxteren et al., 2020). Chatbots have also attracted the interest of practitioners as essential strategic assets for business because they provide the opportunity to offer effective, 24/7 customer care and service (Thomaz et al., 2020). According to Lim et al. (2022), conversational agents can simulate social presence and interaction by presenting humanised images of computers that imitate human abilities, such as recognising and responding to communication, providing feedback and fostering conversation. Due to their accessibility, relative affordability and ease of use for end users, conversational agents facilitate various business processes, particularly those related to customer service (Przegalinska et al., 2019).

By learning from prior discussions and continuously modifying their actions, AI technologies can imitate human conversations and offer more realistic experiences (Mozafari et al., 2021). Whilst chatbots have mostly been used to replace frontline human employees to increase productivity, they inherit the challenges of service interaction quality (Castelo et al., 2023). Convincing customers that chatbots can handle enquiries, provide credible information and deliver reliable services is one of the major challenges of their use (Crolic et al., 2022). Many customers use chatbots primarily to handle straightforward requests, such as finding the answers to frequently asked or routine questions. However, as technology advances, both the range of services that chatbots offer and the expectations of users increase; the inability of chatbots to handle complex conversations remains a significant barrier to their widespread use (Rapp et al., 2021). Preliminary research on negative biases – such as mistrust – associated with the deployment of chatbots also cites customers’ lack of knowledge of procedures (tech literacy) (Syvänen and Valentini, 2020), all of which are elements that prevent customers from utilising the technology (Fernandes and Oliveira, 2021). In summary, though many studies have identified the potential of chatbots in service encounters, none have simultaneously investigated how this technology might affect CX both positively and negatively.

3.1 Case description

Digital technologies and smart services are popular in the retail sector, enabling consumers to visualise products in real time and make more informed purchasing decisions. In this context, our study entails deep into the analysis of an online retail company. Retailers increasingly rely on digital systems for communication (Hänninen et al., 2021). This reliance, coupled with their need for efficient routine actions, drives the adoption of self-service technologies capable of automating customer relations (Sheehan et al., 2020). Consequently, the retail sector presents an ideal setting for analysing both human and non-human interactions, particularly using chatbots. Our focal company, an online menswear store founded in 1985, entered the e-commerce business in 2010. With a customer base spanning Europe, it operates through various distribution channels including offline and online platforms. Before 2020, the company exclusively handled customer service via email and telephone. However, in response to the increasing demand experienced during the COVID-19 pandemic, they implemented an AI-based chatbot in early 2021, developed by an information technology (IT) provider, with the scope to automate conversations on different channels, complement human operators and replicate customer care with automated and personalised responses.

Customers entering the retail company’s website find a button in the lower-right corner with which they can initiate a conversation. By default, the chatbot responds only if prompted, but if the user remains on the same page for an extended time, it presumes indecision and then automatically sends an invitation to start a conversation by asking what the customer needs.

3.2 Data collection

We collected data for 24 months, from January 2021 to December 2022. We collected 7,167 conversations between customers and the chatbot during this time, involving 48,090 messages exchanged. Each customer user exchanged around seven messages with the chatbot; these conversations were not limited to simple requests but were part of longer conversations that consisted of continuous exchanges of questions and answers, similar to conversations between humans. Appendix 1 provides more detailed information about the data, including observations about how customers used the chatbot (how), the types of information to which customer questions referred (what) and the most requested interactions that the chatbot fulfilled (why). Taken together, these data provide both an overall picture of customers’ experiences (relating customers’ intentions to successful responses) and a primary understanding of customers’ interactions with the chatbot (describing customers’ reasonings and behaviours).

Our data capture the times and moments interactions were initiated, as well as transcripts of the actual conversations managed by the chatbot (Hodges et al., 2008). We collected the data in several stages. First, we gathered all 7,167 conversations from the dashboard offered by the online platform that the company used to manage the chatbot. Second, we downloaded and organised similar conversations into a .txt file, to draw connections and abstract meaning from in-depth analysis. The search for conversations ceased when theoretical saturation was achieved, such that no new additional data emerged from reading the conversations (Gummesson, 2017).

We aimed to gain insights into customers’ perspectives and perceptions during their service encounters with the chatbot and understand how the chatbot operated to manage conversations. We sought to comprehend the reasons and times the conversation was initiated, actual positive or negative responses to customers’ needs and accurate executions of the chatbot's programmed tasks. By studying conversations (Hardy, 2001), written by both the chatbot and customers during their interactions and by analysing the language choices (i.e. the use of formal or informal language and emojis), we were able to understand the impact on customers’ overall experiences (Li and Wang, 2023).

3.3 Data analysis using a discourse approach

We analysed the data using discourse analysis, a qualitative, interpretive and constructionist method for examining social phenomena that focuses on the relationships amongst text, discourse and context (Phillips and Hardy, 2002; Mele and Russo Spena, 2021). This technique includes a set of guidelines for conducting structured, qualitative investigations of texts (Wood and Kroger, 2000), as well as a set of presumptions regarding the beneficial effects of language, thus facilitating the acknowledgement of concealed meanings (Burman and Parker, 2016). Discourse analysis allows to acquire data-driven insights from uncommon conversations with chatbots (Wang and Petrina, 2013) as it enables the comprehension of any unintentional intent that may arise during machine learning (Lee et al., 2023).

Discourse analysis views text as the primary data, focussing not on singular content or meaning but rather on explaining how specific ideas or actions come to be expressed and identifying the factors that enable or constrain the language (Cheek, 2004). It stems from the belief that meaning arises from interrelated texts (discourses) that bring new ideas, objects and practices into the world. Discourses are “concrete” in that they are performative in the practices they enact (Phillips and Hardy, 2002, p. 20). In the specific investigated context, this method enables researchers to understand how experiences (i.e., consensus of attitudes, opinions and beliefs held by people) come out, rather than understanding or interpreting phenomena as they currently exist (Hardy, 2001). Discourses are embodied in texts, discourse analysis entails the systematic study of texts to seek evidence of their meaning and determine how this signified meaning translates into new experiences through human-chatbot conversation (Hardy et al., 2004).

We used an interpretative repertoire, which is a collection of words and frequent extracts from conversations with the chatbot. By increasing the possibility of saturation, we also established standards for future analysis that can account for new themes. We identified significant interpretative repertoire nodes, also known as second-order themes, in the initial step of the coding process. The data analysis resulted in the identification of various second-order themes (“parent” repertoires) and first-order themes (“child” repertoires) to outline assorted discursive constructions of customers, in accordance with the coding strategy. Table 1 provides a summary of the interpretative repertoires.

To understand the effect of the chatbots on CX, we conducted a discursive analysis of the collected conversations that allowed us to retrace repetitive elements amongst them, thus deriving emerging themes and their competing interpretative repertoires. We detected seven discursive constructions (i.e. themes) in digital service encounters between the chatbot and customers: interaction quality, information gathering, procedure literacy, task achievement, digital trust, shopping stress and shopping journey. In turn, we identified opposing discourses and define various categories and boundaries that surround the themes and then determine how they formed through the intersubjective perceptions of various actors.

In our study, we achieve validity through performativity, such that the assumptions underlying the analysis, though subjective, support one another and form a coherent theory that we apply to the case study. We obtain reliability through differences in interpretation, which are considered a source of data (Hardy et al., 2004).

4.1 Interaction quality

The exchange of messages between the chatbot and customers affects the CX in terms of interaction quality which refers to the perceived level of quality in the interaction between customers and chatbots within a service encounter (Lee et al., 2021). The chatbot aims to guarantee satisfying experiences and endeavours to resemble a human operator in the words it uses (having a pleasant interaction). Our data analysis shows that through the use of sentences demonstrating that their intentions have been understood, such as “Sure, you have to …” (Conversation #431 - Chatbot message), or gratitude for being able to help, such as “I’m happy I could be of assistance. See you again! ” (Conversation #798 - Chatbot message), the chatbot achieves this goal and enhances experiences. Moreover, the chatbot’s adoption of user-friendly language and emojis is a way in which the technology identifies with human operators and facilitates pleasant interactions with customers: “Cool! Many thanks for your practical and quick assistance! ” (Conversation #3173 - Customer message).

However, if the quality of interactions is insufficient, customers may experience unpleasant moments that have the opposite effect (having an unpleasant interaction). Sometimes, the chatbot’s behaviour is repetitive, because it always responds in the same way or does not understand perfectly. In our data, for example, a customer asks for detailed information on the availability of product sizes. The chatbot, not being connected to the company database, repeats the previous message, says it does not know how to answer, or asks the customer for confirmation that it has understood correctly, without providing a follow-up response. For example: “I’m unable to reply. Perhaps you meant which sizes are offered?” (Conversation #41 - Chatbot message). Customers’ dissatisfaction is evident from their responses, such as: “I have already mentioned several times that I simply needed to know the available sizes. This was just a waste of my time” (Conversation #169 - Customer message). The chatbot’s unfavourable responses lead to unpleasant interactions, prompting customers to leave the conversation.

4.2 Information gathering

The chatbot is a customer resource that enables information gathering, which is the process of collecting data or information to gain insights, knowledge, or understanding about particular needs or issues (Moriuchi, 2023). When customers wish to complete their purchases, or have already done so, assistance may be required if they lack information. The chatbot can offer them help (providing information) coherently and effectively. For example, if a user asks for general information, the chatbot can suggest a page of interest: “To find out more, simply click here to view the sizing table” (Conversation #2834 - Chatbot message), or explain a problem, such as when customers have not received an order confirmation email: “Order and delivery confirmation emails frequently land in spam folders, you can … ” (Conversation #3651 - Chatbot message). In this case, customers respond to the chatbot thanking it for the information received: “Right, it hadn’t crossed my mind to check my spam folder. A thousand thanks!” (Conversation #998 - Customer message).

Sometimes, the chatbot provides inaccurate answers (increasing confusion). It cannot access all of the company’s business intelligence; instead, it is programmed only to respond and suggest certain information. If a customer asks for specific details about a product, the chatbot suggests checking the product sheet but may fail to provide any new information that the customer has not already found: “The product sheet contains further details that could help you” (Conversation #32 - Chatbot message). Similarly, if customers ask for details about their orders or products already purchased, as they often do with human operators, the chatbot cannot retrieve this data and therefore cannot help; sometimes it is even contradictory, leading to other questions, creating frustration and discouraging the customer: “Orders are out of my control; however, you may verify all the details in your personal page” (Conversation #981 - Chatbot message). Customers must deal with conflicting instructions: “Yes, I know, but it doesn’t say anything on my account page” (Conversation #432 - Customer message), thereby increasing their confusion: “I didn’t understand anything, where should I click?” (Conversation #1762 - Customer message).

4.3 Procedure literacy

Through conversations, the chatbot guides customers to follow company procedures. This general theme is also known as procedure literacy, i.e. advancing the spread of company procedures that help customers find easy-to-read information (Smith et al., 2020). The chatbot is readily trained in providing instructions on company policies (knowing procedures) (e.g. shipping, returns). For example, it provides instructions about the payment methods accepted by the company: “Sure. Currently accepted payment methods include PayPal, credit card, bank transfer, and cash on delivery” (Conversation #6588 - Chatbot message). Customers, in response, express their gratitude for the assistance and further engage in the conversation with confidence, seeking answers to specific enquiries, such as: “Thank you, could you also tell me if it is possible to pay in several instalments?” (Conversation #3138 - Customer message).

However, the chatbot is not always able to respond to customers’ needs or explain the correct procedures and certain features or connections that ensure better experiences have been neglected (ignoring procedures). These shortcomings include cases in which customers need to share images with the company; they cannot do so directly during the conversation and can only send them by email: “Sorry, I don’t accept images at this time. You can send it directly to our mail address and we will be happy to assist you” (Conversation #7436 - Chatbot message). Thus, customers leave the conversation without comprehending the usefulness of chatbots: “If I had known this before I wouldn’t have wasted time explaining it to you” (Conversation #7436 - Customer message). Failures also occur when customers need specific information (e.g. updates on shipment status), which is not provided directly by the company but rather requires input from a third-party actor. In this case, the chatbot ignores the procedure by providing simple information about where to find the tracking code, without providing any information relating to the request: “Using the tracking number from the email confirmation, you can monitor the status of your order directly on the courier’s website” (Conversation #345 - Chatbot message). Customers, in turn, respond, “I can’t find the email, I’m looking for a quicker solution ” (Conversation #22 - Customer message).

4.4 Task achievement

The theme of task achievement pertains to how effectively a chatbot guides users through conversational flows and assists them in accomplishing their tasks or goals (Roy and Naidoo, 2021). The chatbot, as a form of technology that autonomously handles conversations, attempts to identify customer intentions and respond appropriately with contextual messages (improving tasks): “By clicking on a product, you can easily select from a range of available sizes and colours ” (Conversation #918 - Chatbot message). In this way, it allows customers to self-manage the switch between autonomy and assistance, offering them experiences suited to their needs, quickly when possible and facilitated when necessary. This capacity is highly valued by customers, as they confirm: “Thanks, I did it. However, I would like to pair them with trousers, could I speak to an operator for advice?” (Conversation #769 - Customer message). Thus, the chatbot also enables multitasking, because it performs interconnection actions with the operator whilst continuing to answer other questions: “While an operator is connecting, can I help you in any other way?” (Conversation #1111 - Chatbot message).

However, some tasks can be managed badly, worsening the customer’s experience (worsening tasks). If the chatbot does not understand the customer’s intentions, it does not know how to act or how to respond. Its response of “I’m not sure how to respond to this question” (Conversation #90 - Chatbot message) or “This question is beyond my capacity to respond” (Conversation #1066 - Chatbot message) contrasts with the very reasons for its design and implementation in corporate activities. In turn, customers experience disappointment and frustration and leave the conversation without adequate assistance: “Ugh, that’s frustrating!!! I was counting on you to be able to help me ”. Unsatisfactory behaviours or responses by the chatbot can diminish customer perceptions of the company’s reliability and competence.

4.5 Digital trust

The reliability of the system involves a shift from physical to digital and human to machine, which we refer to as digital trust, reflecting users’ positive beliefs about accepting and using voice-assisted AI systems (Fernandes and Oliveira, 2021). Chatbots foster positive emotional states when customers are confident about the company (increasing trust), such as when the chatbot reassures them with comforting words: “It’s not a problem, sometimes it happens” (Conversation #21 - Chatbot message). The chatbot can explain that a shipment is late and that employees are doing everything they can to resolve the problem: “The current health emergency may cause delays in order processing and shipping. We apologise for the trouble! ” (Conversation #1 - Chatbot message). Customers seem to appreciate this reassuring behaviour: “Thanks for the heads-up, much appreciated! ” (Conversation #103 - Customer message).

However, if the chatbot does not provide such reassurance or does so incorrectly, customer dissatisfaction increases, worsening an already difficult change process (reducing trust). In this case, not understanding customers’ questions and answers, providing information that differs from that previously communicated by a human operator because of slow updating, or simply replying to customers that the operators are busy and that they must wait likely reduces trust even further: “Okay, hold on a second. Although all operators are currently busy, they will be with you as soon as possible!” (Conversation #69 - Chatbot message). Customers value their time and expect a prompt resolution from the chatbot. If it fails to meet their expectations, their trust begins to wane. They might express this as follows: “Next time I’ll call directly. You’re of no use!” (Conversation #69 - Customer message).

4.6 Shopping stress

The state of anxiety that accompanies customers’ purchasing actions is known as shopping stress, and this negative psychological reaction can arise when a shopper feels overwhelmed by certain circumstances or situations (Lucia-Palacios et al., 2020). By using a chatbot that guarantees the achievement of a series of tasks quickly and automatically, whilst maintaining human features, the company can develop better experiences. Increasing the speed of interactions and asking customers for more information to avoid misunderstanding can reduce customer stress (reducing stress). This behaviour is intrinsic to the chatbot; it responds only in this way if it achieves a particular degree of certainty and otherwise asks for more details: “I’m not sure what you’re asking, but might this possibly be what you mean? Payment methods or refund methods” (Conversation #11 - Chatbot message). Customers highly value the chatbot’s ability to provide clear information, which reduces shopping stress and encourages them to continue the conversation: “Finally, thanks!!! If I need further information, I will contact you ” (Conversation #89 - Customer message).

To try to improve the CX, the company programs text that indulges customers by apologising for their waits or suggesting they make themselves comfortable whilst waiting: “While you wait you can have tea, coffee, or ask me if you need anything else” (Conversation #4568 - Chatbot message). However, chatbot use can also easily lead to unpleasant episodes, which have the complete opposite effect on the CX. If the chatbot does not understand, cannot respond, or does not have the information needed to assist customers promptly, this only adds to shopping stress (increasing stress). If the user is in a hurry and thinks the chatbot will respond quickly, and then it fails to do so, the failure will fuel the customer’s tension. For example, our data reveal counterproductive effects when customers repeatedly ask for an operator’s assistance or degrade the chatbot’s functioning: “You are not useful” (Conversation #815 - Customer message) or “you are a very pathetic robot” (Conversation #10 - Customer message).

4.7 Shopping journey

As a technology that interacts with customers, the chatbot is inserted in different moments of the shopping journey, referring to the stages a customer goes through from pre-transaction to transaction to post-transaction (Hoyer et al., 2020). From an experiential point of view, being able to complete the journey is a fundamental element of the main themes that influence CXs. When customers initiate conversations with the chatbot and then the chatbot, through questions and suggestions, leads to eventual need satisfaction, the shopping journey is completed (completing the journey stage). In the conversations we analysed, the chatbot accompanied customers during their entire journeys, such as by showing them easier routes to check out: “To complete the transaction, you just click on confirm without accessing any additional pages ” (Conversation #5541 - Chatbot message). Customers were satisfied with the conversations and the chatbot’s answers, expressing gratitude through comments such as: “Thanks for your support ” (Conversation #3324 - Customer message) or “See you soon ” (Conversation #5171 - Customer message).

However, in as many conversations, customers end the interaction by demonstrating impatience and frustration. If after one or more messages, they do not receive the correct answers, customers often end the conversations and close the site (abandoning the journey stage). For example, customers say, “Yeah. Ok whatever ” (Conversation #4933 - Customer message). Their experience is even worse if the chatbot adds nothing new and tells customers they must verify their orders themselves: “Sorry again. Currently, no operators are available. Come back later!” (Conversation #2212 - Chatbot message). In such situations, conversations end in the same way, through dissatisfaction – “Fair enough, forget it. Take care” (Conversation #2871 - Customer message) – or even abruptly with no response from the customer.