5 min read

Containment is the Wrong Metric! Here’s Why

Are you locking out customers?

Ever get the feeling something’s just too easy? That’s how we feel when we’re looking at containment — the number of calls or chats that don’t go to an agent, divided by the total number of calls or chats received. Containment is the industry standard for measuring IVR and chatbot performance. But should it be?

Are you sure more containment is really what you want?

Containment is a misleading metric. Business managers may think their IVR and chatbots are performing very well if they see containment going up. But containment can be both good and bad. If you’re just looking at a simple percentage, you’re not getting the full story.

What People Think About in Terms of Containment…

It occurs when a customer calls up an IVR system or chats with a chatbot and fully resolves their query through those channels—without ever speaking to a customer service representative. They complete their payment, receive the necessary text confirmation, and put down their phone or close their chat screen. Then they leave satisfied, knowing everything’s been done properly.

This is what we want, right? Your human agents weren’t held up for several minutes on a call that could have been easily handled by your bots. Your customer didn’t wind up holding in a queue for 10 minutes. Everyone’s happy, everything’s cool. Perfect.

Do you feel a “but” coming?

…Versus the Reality of Containment

There are many scenarios that you don’t want playing out in your contact center, but they can contribute to higher containment scores! This is what people tend to forget about. When businesses blindly follow containment, it can lead to negative outcomes.

If you see a call or chat is contained, don’t take it at face value. It may have played out like the rosy picture we painted above. Or it could have been the exact opposite scenario. Sometimes a call or chat is contained because it so thoroughly fails to meet a customer’s needs. If the experience is bad enough, the customer may drop off entirely. That’s a bad thing. Yet from a pure metrics standpoint, it looks like a win. After all, the call was contained, right?

Bad containment occurs when:

  • The system doesn’t understand the customer, and the interaction goes round and round until the (rightly) aggravated customer hangs up.
  • The customer doesn’t understand the options the system presents, and the same thing happens. Frustration takes root, and the customer’s patience runs out.
  • A backend problem occurs and the system actually tells the customer something along the lines of “I’m sorry, but I can’t help you right now,” and the call ends.

All of these scenarios will still cause your containment percentage to go up. However, they hardly call for celebration.

It’s not just good or bad

Containment can be neutral, too. Neutral containment may occur when someone simply dials the wrong number. They reach your IVR system, realize their mistake, and hang up. No harm, no foul, right? Sure—as long as you’re paying attention and not putting this in your wins column.

Neutral containment shows how bogus a metric like containment is in isolation. Is neutral containment as harmful to your business as negative containment? Maybe not, but it does further muddy the waters and inflate your metric.

Something as simple as a misplaced credit card or a child calling for their parent may also result in neutral containment. The customer is in the right place, but life gets in the way and they’ll try back later. Again, that’s not creating a negative experience that can lose you a customer. But it is providing false insights into how your business is operating.

100% Containment?

We hear customers talk about improving containment all the time. When they do, we’re happy to share our insights on why containment by itself is meaningless. We explain how bundling the good, the bad, and the neutral together really doesn’t make much sense.

Nothing really highlights the absurdity of this better than the 100% containment example. That’s because achieving 100% containment is simple: Unplug your IVR and hang up on people or just automatically log them out of their chat session.

This always gets a laugh from our customers—but it also shows just how misleading the containment metric can really be when viewed in a vacuum. No one wants to hang up on their customers, obviously. But framing it this way helps to show how misleading containment can be.

You Need to Dig Deeper

We’ve been a little hard on containment here, but it comes from a good place. We have over 20 years of experience in the conversational AI field, and we’ve seen businesses misled by containment metrics too many times to just let it slide. Understanding containment properly can yield some valuable insights, but you need to dig deeper to uncover exactly what your containment metrics are trying to tell you.

Even focusing on great containment and taking those junk wins out of the equation leaves blind spots, such as:

  • Partial automation. In our experience, we’ve seen as much as a third of businesses’ financial benefit come from partial automation—not complete. This is why task performance is so important. You may not contain a call completely. But if your bots are able to identify and verify a customer before that customer goes to a human agent, you’ve automated two of the tasks that need completing. The customer is happy because they’ve accomplished their desired task. The business is happy because they’ve saved time. The call or chat wasn’t contained, but it was a smooth experience for all involved.
  • Routing to the right agent. Sometimes a customer simply needs to speak to the right agent with the right skills. Bots are capable of excellent work within their scope of design. But human agents have different skill sets, and freeing up their time through partial automation lets them focus on more beneficial areas. If you’re hyper-focused on containment, you may see a call or chat routed to an agent as a failure or inefficiency. In reality, it could be exactly the right thing to do.

Beyond Containment: How to Accurately Measure Your IVR or Chatbots’ Success

Task Performance

Focusing on the individual tasks your IVR or chatbots handle for customers is the key to understanding containment. The simple containment percentage is useless. However, there are three metrics within each individual task that paint a much fuller picture of how your bots are performing.

  • Task complete: The task was automated and completed as expected.
  • Task failure: The task was not completed due to back-end issues or because the system wasn’t able to understand the customer
  • What we call “flow complete”: The system completed the task as expected, but the transaction was not actually completed. The customer’s credit card may have been declined, or they were prevented from cancelling the order because it had already been dispatched. The bot did everything a human agent could have done.

Customer Experience Metrics

Finally, you need to look at customer experience metrics. It is easier to retain a customer than to acquire one. Your IVR or chatbot’s job is not to contain as many contacts as possible but rather to handle those customer enquiries that can best be served by bots, and redirect calls that require human attention to the right agent as quickly as possible. That way you’re optimizing the customer experience and contact cetner costs at the same time.

Important customer experience metrics include:

  • CSAT (Customer Satisfaction Score) – how satisfied are your customers with your business?
  • NPS (Net Promoter Score) – how loyal are your customers to your business?
  • LTSA (Likelihood to Shop Again) – how likely is a customer to shop with or purchase from your business again?

So think beyond containment. Dig into task performance and consider mechanisms like post-call or transactional surveys to understand what your customers think (and check out our blog on the difference between CSAT and NPS). The more you know about your customers and their experience with your IVR or chatbots, the better. And you’re not going to get that from looking solely at containment. If you’re struggling to understand how your IVR and chatbots are performing, our Bot management service can provide the insights you need.


4 min read

How Do I Get More ROI Out of My IVR, Voice, or Chatbot Platform?

Getting more out of your IVR and chatbots

You’re expected to meet escalating customer experience expectations and manage costs at the same time. Conversational AI systems like IVR, voice, and chatbots are supposed to help, but often the outcomes don’t live up to the promise. So how can you maximize the ROI from your Conversational AI?

It’s tempting to turn to your tech team or vendor and demand better results, or look for a new platform that will solve the problem. But the platform is just that: it’s the foundation you build on. It’s what you do with that platform that nets your return.

In this article, I’ll show you how to spot the problems that might be affecting your bot workforce, and some simple techniques to coach and help them improve. Just like you would with your own team.

How Problems With Your Bots Affect Your ROI

Everybody says they want a better customer experience and lower costs. But how do you get that? The answer is to get the right team members, bots and humans, doing the right things and working together seamlessly.

The goal of your contact center is to get customers to a skilled agent who knows how to help, or a bot that can do the job just as well. Your bot workforce and your human workforce need to work together to achieve that goal. For your human agents, you’ll routinely spend time listening to their conversations with customers, watching their performance metrics, and coaching and training them to do better. You need to do the same for your bots.

What’s Causing Problems With My Bots to Begin With?

Bots can do exceptional work in the specific areas they’re designed to operate. But four organizational issues can make things really difficult for your bot:

  1. Lack of maintenance and updating. Just like your human workforce, you need to keep your bots up to date. Let them know when a new product launches, or a process changes. Your bots won’t figure these things out on their own!
  2. Uncontrolled changes. Conflicting instructions will undermine the confidence and capability of your agents. Same with bots. You wouldn’t let anyone and everyone from across the business tell your agents what to say. You need to apply the same standard to your bots. Uncontrolled changes have unintended consequences.
  3. Technology problems. Sometimes, your agents can work around problems in the CRM. Bots can’t. So watch out for changes in the backend APIs that will have Bots spouting garbage to your callers.
  4. Poor conversation design. You need to bring in an experienced conversation designer to address this issue.

Chances are that some or all of these problems are already undermining the performance of your bot workforce, so how do you figure out what needs fixing?

Apply Your Existing Management Skills to Your Bot Workforce

Among your skills as a contact center or business manager is the ability to listen to and work with your human agents to uncover problems and cultivate your team’s skills. You know how to hold effective one-to-ones to identify areas of improvement as well as growth opportunities.

Just as you work closely with your human agents to optimize their performance, you can work closely with your voice, IVR, and chatbots to do the same—with a few tweaks. Your bots can’t tell you what problems they’re having. So how do you apply your existing skills to effectively manage your bots?

  1. Listen to call recordings and read chat transcripts. With a human agent, you’d probably just do side-by-side call listening or chat monitoring. With your IVR, voice, and chatbots, listening to their interactions with customers and reading chat transcripts serves the same function. Get your tech team to pull a selection of whole call recordings, or chat transcripts, so you can review the interactions your bots are having with your customers. You’ll learn a lot.
  2. Mystery shop your bots. Mystery shopping is a great way to put yourself in the shoes of your customer and experience what they experience. You can do the same with your IVR or chatbots. Just call up and pretend to be a real customer. To really get a feel for things you’ll need to have an end customer account or actually go buy and return an item to see how your bots handle things.
  3. Dig into data. Your bots can’t tell you what they’re struggling with, but you can learn a lot from digging into the data. Are you getting a lot of internal transfers? That could mean there’s a routing problem. Or are people dropping off the call or closing the chat session mid-task? Could mean the bot is getting stuck. When you see these problems, go back to the call recordings, chat transcripts, and mystery shopping to understand what’s really going on.

You’ve Identified the Issues with Your Conversational AI. Now What?

You don’t need to be a techie or a conversation design expert to spot problems in your bot workforce. Just treat them like you would a human workforce. You know what good performance looks like. You know the kind of experience your customers expect and you want to deliver. And hopefully this article will help you see where your bots are falling short. But how do you fix the problems you find?

If your bots are basically doing the right things, but not understanding the customer or being understood themselves, the answer could be to update your bots with a better conversation design. But often the problems are deeper. You wouldn’t hire a bunch more agents without a strategy for what they’ll do, what skills they’ll need, and how you’ll measure and manage their performance. Same goes for bots. If the problems run deeper, you probably need to build a better Bot Strategy.

3 min read

AI can’t solve your customer service problem. Here’s why.

AI can’t solve your customer service problem. Here’s why.

Everyone’s talking about AI. Like ‘the cloud’ numerous years ago, if your product isn’t powered by AI forget it. If you believe everything you read you’d think AI is the answer to everything from driverless cars (which it is) to natural and engaging chatbots, intelligent assistants and IVR systems (which, unfortunately, it isn’t).

4 min read

Conversational AI and the Future of Customer Service?

Conversational AI and the Future of Customer Service?

Are intelligent conversational AI assistants like Siri and Alexa the next big thing in customer service? It’s starting to look that way – but the customer experience has to be right.

5 min read

Why Callers are Falling Out of Love with Telephone Customer Service

Why Callers are Falling Out of Love with Telephone Customer Service

New research from the UK’s Institute of Customer Service says more customers rely on telephone service than anything else – but that doesn’t mean they’re happy.

4 min read

Don’t Lose Customers Over a Prolonged Omni-Channel Adoption

Don’t Lose Customers Over a Prolonged Omni-Channel Adoption

Switching to omni-channel customer service may seem like a major undertaking – but big plans on the horizon are no excuse for failing to make valuable improvements to your IVR now.