How To Conduct an A/B Test on IVR Voice Applications

There is no single way to assess the effectiveness of your voice applications. For people who are new to voice automation or VoiceXML it may not be clear where to start with A/B analysis.

Of course, this begs the question: “why bother?” After all, you have your application up and running, and it works fine, right? Just because it works well from your point of view, doesn’t mean it works well for your customers. Taking the extra step or two to make sure your application provides a good customer experience could be the difference between keeping or losing a customer.

A recent post here provided some code that people could use to build a self-service app to check an account balance, and suggested a number of areas that would be good options for A/B analysis. The way your company structures its voice-based customer service can have an impact. Having the right tools at your disposal definitely helps, too.

Simultaneous or Linear Testing

The number of phone lines you have feeding your customer service can impact the way you A/B test. If you only have two or more phone lines, it’s possible to clone your application and deploy different apps to different numbers. This is a more difficult approach, but it can be done. The benefit to this is that you can do a traditional A/B test where your different app variations are running simultaneously.

Because telephony is so complex, a more linear, iterative approach tends to work better. Deploy your application and start collecting data on how it performs. Once you have a baseline, you can then make changes to the application in production, collect data on that iteration for a specific period of time, and then compare those results to the previous iteration.

If the results are what you wanted to see, then you can keep the changes and test something else. If they’re not what you expected, then it’s easy to revert the application back to the previous iteration and try something different. Remember, it’s best to make changes incremental, one variable at a time.

Now it’s worth remembering that the linear approach works just as well for an application with multiple lines feeding into it. Depending on the size or structure of the system in question, that may be preferable.

Determining What Works

Fortunately, for Plum Voice customers, they have access to VoiceTrends, which is an analytics toolkit built specifically for voice communications. With a whole host of data at your disposal the you then have to figure out what areas you want to improve.

Some common things to think about include:

  • Average Call Time
  • Transfer Rate
  • Average Number of Forms Fill Out
  • Average Number of Loops Per Call

All of these data, and others, tell you how callers interact with your voice applications. To aid in this process, VoiceTrends includes a diagnostic flow feature that visually presents the user behavior based on real data.

For example, if you want to reduce the transfer rate so fewer people are opting out of self-service for customer service representatives, these data will show you exactly where people are transferring out of the application.

If you want to improve your application’s transfer rate you need a baseline. The graph here provides that. Let’s say you want to keep your transfer rate below 10% at peak usage times. The troughs in this data fall on the weekends (e.g., August 13). Lighter usage on these days skews your overall transfer rate lower.

VT AB Test Screenshot

The reason callers leave the self-service application may be because the prompt for a particular menu is too long, or that the option people want is far down the menu list and they don’t want to wait for it. These things can trigger the old forget-it-and-press-0-for-the-operator move.

To get a visual representation of how callers use your voice application check out the diagnostic flow in VoiceTrends. This details all of the transfers that occur within the application. Calls start at the main menu, which gives users the choice between customer service or sales. Billing is located under the customer service menu. (Note, the VoiceTrends diagnostic flow is a dynamic, interactive tool. Hovering over an item displays additional information about that interaction.)

Looking at this call flow, you might want to make billing a higher priority. Once people get to the billing menu they tend to have successful transactions. The billing menu isn’t immediately obvious however, as it’s buried in the customer menu. If you want to put a greater emphasis on payments, then a worthwhile test here might be to move the billing menu out from the customer menu and put it in the top menu with the sales and customer options.

After making this change in your application and collecting data on the new configuration, you can better determine if moving billing to the top menu improved your transfer rate.

If you want more VoiceTrends examples be sure to check out an earlier blog post that dives deeper into how to use VoiceTrends analytics to solve voice communications problems.

It’s helpful to remember that no application will be perfect from the word go and to get the best results—and happier customers—monitoring, updating, and optimizing applications should be an on-going process.

(Visited 470 times, 1 visits today)