Emerging TechnologyChatbotsChatting with bots: How well 5 brands’ chatbots actually work

Chatting with bots: How well 5 brands' chatbots actually work

We usually write about what chatbots can do, rather than how well they do it, so we're testing branded bots from the likes of Sephora, eBay, and H&M.

We write about chatbots all the time and how could we not? Global market revenue was more than $250 million last year. That number is only going up; Gartner predicted that by 2020, 85% of business will be conducted without human interaction.

Brands consistently find ingenious ways to improve their customer experiences with the assistance of artificial intelligence. Early-adopter Sephora makes product recommendations with a chatbot. Starbucks’ artificially intelligent barista takes your order.

While we frequently write about chatbots, we tend to skim the surface, covering their capabilities without really looking under the hood. Sure, Sephora’s chatbot can answer your questions—but can it do so as well as a person? On paper, self checkout certainly should get customers in and out of a store faster than a human cashier, but that’s not necessarily the case. Have you ever noticed how grocery stores always seem to have a cashier hanging around those machines to handle the inevitable troubleshooting?

In March, PricewaterhouseCoopers found that 75% of people around the world prefer interacting with people over bots. Does that mean consumers (outside of Japan) are just slow to get on board? Or are brands overestimating how much they facilitate a better, more seamless customer experience? We decided to revisit five chatbots we’ve written about previously, just to see how well they do their jobs.

PwC - People over bots

We decided to revisit five chatbots we’ve written about previously, just to see how well they do their jobs.

1. Sephora

On Kik, Sephora’s bot opens up with an invitation to take a style quiz, which helps deliver the most relevant product recommendations. The questions were all multiple choice and none of the answers fit in the “I’m just here to have conversations with chatbots for a story” category, so I answered them as if I were a real shopper. The chatbot got to the bottom of what I was allegedly looking for, serving me product recommendations, complete with reviews, within seconds.

It’s certainly helpful, but only if you stick to the script and use the quick replies. The bot has difficulty with variations in wording, though the answers nail the brand voice.

Chatbots- Sephora

2. eBay

A chatbot can come in handy on eBay, a website that’s overwhelming by nature, given that its inventory consists of literally anything. During the height of the 2000 boy band craze, a radio station auctioned off Justin Timberlake’s half-eaten French toast and someone bought it for more than $1,000. I went the more realistic route and looked for shoes. I also decided to include a few extraneous details just to see if eBay’s ShopBot on Facebook Messenger could keep up. It could, skipping over them and getting right down to business.

ShopBot’s initial recommendations are a mixed bag, but once you select one, it quickly hones in on what you’re looking for. The bot also has such a conversational tone and an impressive ability to answer unscripted questions, it’s easy to forget you’re talking to a machine. It even schmoozes (“You’ve got great taste”) like a real sales associate.

Chatbots- eBay

3. Bank of America

In March, Bank of America launched Erica, the virtual assistant who lives within the bank’s app. She’s certainly preferable to the human customer service representatives, who are perfectly helpful, though generally hidden behind several long, unskippable voice menus.

AI has come a long way from the early versions of Siri, who only understood you if you rea-lly ov-er-en-unc-i-ated. Erica has no problem with voice commands and is very helpful when it comes to basic actions, such as balance checks and transfers. I’m looking to refinance my student loans and I don’t know if Bank of America does that. Erica doesn’t, either.

Chatbots- Bank of America

4. H&M

This isn’t the first time we’ve written a story based on conversations with branded bots, but given how quickly technology the landscape evolves, we thought it was worth revisiting. Last June, Chris Camps found H&M’s bot on Kik clunky and I have to say, I agree.

Foundationally, it’s similar to the Sephora chatbot, trying to discern your tastes in order to serve the best recommendations. However, it’s a bit too specific, asking shoppers to choose between items, rather than categories. The bot’s outfit builder serves you products at random and if you’re not looking for sweatpants, well, too bad.

Chatbots- H&M

5. Kayak

I don’t understand why Kayak has a chatbot. Unlike Alexa, Facebook Messenger requires typing, so you’re essentially putting your query in the app instead of the far superior website. There, you can find a find a flight with little fanfare. Enter the basics and your search results come with filters: time, airport, whether it’s nonstop.

On Messenger, the same information requires far more steps, since the bot asks you questions one at a time. It’s also fairly unsophisticated. When I searched for a flight from LaGuardia, Kayak assumed I meant the town of 1,450 in Spanish Basque country, not the major airport three miles from where I’m writing this. Once we got past this snafu, the chatbot still didn’t show me any flights because it interpreted “Monday” as right now, two hours before the clock technically strikes Tuesday.

Chatbots- Kayak

Conclusion

My (admittedly small sample size of) chatbot experiences were a mixed bag. Using Kayak was aggravating, but a robot also almost talked me into buying a pair of shoes. If they had my size, they’d be at my apartment faster than you could fly to Laguardia, Spain. Somewhere in the middle, I could do basic banking transactions, but not look into student loan refinancing.

In Bank of America’s defense, that was a trick question. The bank doesn’t offer that service, but even if it did, I wasn’t actually looking to refinance my student loans with a chatbot. I don’t imagine most other people would, either. That’s not what it was made for.

We may interact exclusively with chatbots in the future, but we’re certainly not there yet. Chatbots are there to complement the customer experience, not dominate it. And in that regard, they work exceptionally well. Erica amassed 1 million users in three months. According to MarTech Today, Sephora customers who use the chatbot spend $50 more than those who don’t. (And they’re also pretty fun to use.)

Related Articles

Bots, chatbots, robots, AI! Here’s why knowing the difference could set your company apart

AI Bots, chatbots, robots, AI! Here’s why knowing the difference could set your company apart

10h Emily Alford
Ready to talk to robots? Natural language processing is changing search and customer service

Chatbots Ready to talk to robots? Natural language processing is changing search and customer service

3w Emily Alford
Chatbot market size is set to exceed USD 1.34 billion by 2024

Chatbots Chatbot market size is set to exceed USD 1.34 billion by 2024

2m Rahul Pise
5 brands that prove chatbot-powered marketing is the future

Chatbots 5 brands that prove chatbot-powered marketing is the future

3m Emily Alford
8 ways AI can enhance your marketing strategy today

AI 8 ways AI can enhance your marketing strategy today

4m dinesh
6 tips for building compelling chatbots (even if you're not a programmer)

Chatbots 6 tips for building compelling chatbots (even if you're not a programmer)

11m Marcela De Vivo
Which industries have the greatest potential for chatbot disruption?

Chatbots Which industries have the greatest potential for chatbot disruption?

11m Al Roberts
7 reasons not to believe the chatbot hype

Chatbots 7 reasons not to believe the chatbot hype

1y Al Roberts