Ever get the feeling something’s just too easy? That’s how we feel when we’re looking at containment — the number of calls or chats that don’t go to an agent, divided by the total number of calls or chats received. Containment is the industry standard for measuring IVR and chatbot performance. But should it be?

Are you sure more containment is really what you want?

Containment is a misleading metric. Business managers may think their IVR and chatbots are performing very well if they see containment going up. But containment can be both good and bad. If you’re just looking at a simple percentage, you’re not getting the full story.

What People Think About in Terms of Containment…

It occurs when a customer calls up an IVR system or chats with a chatbot and fully resolves their query through those channels—without ever speaking to a customer service representative. They complete their payment, receive the necessary text confirmation, and put down their phone or close their chat screen. Then they leave satisfied, knowing everything’s been done properly.

This is what we want, right? Your human agents weren’t held up for several minutes on a call that could have been easily handled by your bots. Your customer didn’t wind up holding in a queue for 10 minutes. Everyone’s happy, everything’s cool. Perfect.

Do you feel a “but” coming?

…Versus the Reality of Containment

There are many scenarios that you don’t want playing out in your contact center, but they can contribute to higher containment scores! This is what people tend to forget about. When businesses blindly follow containment, it can lead to negative outcomes.

If you see a call or chat is contained, don’t take it at face value. It may have played out like the rosy picture we painted above. Or it could have been the exact opposite scenario. Sometimes a call or chat is contained because it so thoroughly fails to meet a customer’s needs. If the experience is bad enough, the customer may drop off entirely. That’s a bad thing. Yet from a pure metrics standpoint, it looks like a win. After all, the call was contained, right?

Bad containment occurs when:

• The system doesn’t understand the customer, and the interaction goes round and round until the (rightly) aggravated customer hangs up.
• The customer doesn’t understand the options the system presents, and the same thing happens. Frustration takes root, and the customer’s patience runs out.
• A backend problem occurs and the system actually tells the customer something along the lines of “I’m sorry, but I can’t help you right now,” and the call ends.

All of these scenarios will still cause your containment percentage to go up. However, they hardly call for celebration.

It’s not just good or bad

Containment can be neutral, too. Neutral containment may occur when someone simply dials the wrong number. They reach your IVR system, realize their mistake, and hang up. No harm, no foul, right? Sure—as long as you’re paying attention and not putting this in your wins column.

Neutral containment shows how bogus a metric like containment is in isolation. Is neutral containment as harmful to your business as negative containment? Maybe not, but it does further muddy the waters and inflate your metric.

Something as simple as a misplaced credit card or a child calling for their parent may also result in neutral containment. The customer is in the right place, but life gets in the way and they’ll try back later. Again, that’s not creating a negative experience that can lose you a customer. But it is providing false insights into how your business is operating.

100% Containment?

We hear customers talk about improving containment all the time. When they do, we’re happy to share our insights on why containment by itself is meaningless. We explain how bundling the good, the bad, and the neutral together really doesn’t make much sense.

Nothing really highlights the absurdity of this better than the 100% containment example. That’s because achieving 100% containment is simple: Unplug your IVR and hang up on people or just automatically log them out of their chat session.

This always gets a laugh from our customers—but it also shows just how misleading the containment metric can really be when viewed in a vacuum. No one wants to hang up on their customers, obviously. But framing it this way helps to show how misleading containment can be.

You Need to Dig Deeper

We’ve been a little hard on containment here, but it comes from a good place. We have over 20 years of experience in the conversational AI field, and we’ve seen businesses misled by containment metrics too many times to just let it slide. Understanding containment properly can yield some valuable insights, but you need to dig deeper to uncover exactly what your containment metrics are trying to tell you.

Even focusing on great containment and taking those junk wins out of the equation leaves blind spots, such as:

• Partial automation. In our experience, we’ve seen as much as a third of businesses’ financial benefit come from partial automation—not complete. This is why task performance is so important. You may not contain a call completely. But if your bots are able to identify and verify a customer before that customer goes to a human agent, you’ve automated two of the tasks that need completing. The customer is happy because they’ve accomplished their desired task. The business is happy because they’ve saved time. The call or chat wasn’t contained, but it was a smooth experience for all involved.
• Routing to the right agent. Sometimes a customer simply needs to speak to the right agent with the right skills. Bots are capable of excellent work within their scope of design. But human agents have different skill sets, and freeing up their time through partial automation lets them focus on more beneficial areas. If you’re hyper-focused on containment, you may see a call or chat routed to an agent as a failure or inefficiency. In reality, it could be exactly the right thing to do.

Beyond Containment: How to Accurately Measure Your IVR or Chatbots’ Success

Task Performance

Focusing on the individual tasks your IVR or chatbots handle for customers is the key to understanding containment. The simple containment percentage is useless. However, there are three metrics within each individual task that paint a much fuller picture of how your bots are performing.

• Task complete: The task was automated and completed as expected.
• Task failure: The task was not completed due to back-end issues or because the system wasn’t able to understand the customer
• What we call “flow complete”: The system completed the task as expected, but the transaction was not actually completed. The customer’s credit card may have been declined, or they were prevented from cancelling the order because it had already been dispatched. The bot did everything a human agent could have done.

Customer Experience Metrics

Finally, you need to look at customer experience metrics. It is easier to retain a customer than to acquire one. Your IVR or chatbot’s job is not to contain as many contacts as possible but rather to handle those customer enquiries that can best be served by bots, and redirect calls that require human attention to the right agent as quickly as possible. That way you’re optimizing the customer experience and contact cetner costs at the same time.

Important customer experience metrics include:

• CSAT (Customer Satisfaction Score) – how satisfied are your customers with your business?
• NPS (Net Promoter Score) – how loyal are your customers to your business?
• LTSA (Likelihood to Shop Again) – how likely is a customer to shop with or purchase from your business again?

So think beyond containment. Dig into task performance and consider mechanisms like post-call or transactional surveys to understand what your customers think (and check out our blog on the difference between CSAT and NPS). The more you know about your customers and their experience with your IVR or chatbots, the better. And you’re not going to get that from looking solely at containment. If you’re struggling to understand how your IVR and chatbots are performing, our Bot management service can provide the insights you need.