≡ Menu

A culture of experimentation

Experimentation is an essential ingredient in Digital Decisioning and has been identified as a top technology-enabled business trend:

What if you could analyze every transaction, capture insights from every customer interaction, and didn’t have to wait for months to get data from the field?… many companies are taking data use to new levels, using IT to support rigorous, constant business experimentation that guides decisions and to test new products, business models, and innovations in customer experience.

The McKinsey Quarterly (Bughin, Chui, & Manyika, 2010)

Experimentation is particularly important when considering operational or micro-decisions, which are the transactional, customer-specific, small-scale decisions at the core of Digital Decisioning. Operational decisions and experimentation are tightly linked because:

  • Strategic decisions are often those about choosing among alternative approaches. Each approach can be an experiment, but it is the operational decisions that are being made differently in each experiment. For instance, the strategic choice might be to retain and develop customers rather than to acquire new ones. To experiment, you need to be able to vary both the customer retention and customer acquisitions you make every day. You cannot conduct the kind of experimentation being discussed unless you have control of these micro-decisions, that is, unless you have implemented Digital Decisioning.
  • Because operational decisions are repeated over and over, the value of experimentation is higher. If an experiment shows that one of two approaches works better, then there will be more operational decisions to be made in the future, and you will be able to apply the approach that worked better to those decisions.
  • Predictive analytic and machine learning models do not make definitive predictions, rather, they turn uncertainty into a usable probability. Because the predictions are not completely definitive, you will have to make choices about how to use those predictions. Being able to experiment with different approaches will show you how to use the predictions to improve your results.
  • Many organizations don’t have the data they need to define an effective decision-making approach or to build a predictive analytic model. Sometimes the only way to capture the data you need is to run experiments to see what happens in different circumstances. In particular, organizations need to run experiments to fill in gaps where certain actions are never taken with certain customers. If customers above a certain risk level are never accepted, then there will be no data about the behavior of these kinds of customers unless an experiment is explicitly designed to capture it.

When you implement Digital Decisioning, you need to build in the capability for experimentation. This requires that both IT and business teams become more comfortable with experimentation. It also means that you need to develop a deep-seated awareness of the importance of control groups and a basic understanding of experimental design.

IT Experimentation

IT organizations have never really been able to build experimentation into a system. Legacy development approaches don’t lend themselves to supporting experimentation and the rapid evolution that goes with it. In addition, most IT organizations work hard to find the “right” answer for any given problem and then implement it. They want to work with their business partners and apply their own expertise to find the right answer so they can code it and hand it over to their business users. Experimentation is very important to Digital Decisioning, so this can represent a challenge.

It can be difficult to persuade IT organizations that it is not always possible to know the right answer. Furthermore, the right approach, even if it could be determined now, is likely to cease being the right approach at some (unknown) point in the future. For Digital Decisioning, this point may not be very far in the future either, as the pace of change is often very high in decision-making. Accepting that the system should, therefore, constantly challenge the approach believed to be the best one to see when it begins to fail is also a big change.

Digital Decisioning that is not 100% based on regulations and fixed policy must allow for multiple approaches to be tried. What is worse is that most of the alternatives tried will, in fact, underperform the default. The IT organization must accept that building this capability into the system anyway has long-term value for Digital Decisioning. There is no particular trick to this other than to build on the collaboration that is so important for Digital Decisioning in order to help them understand how Digital Decisioning is different from their past experience.

Business Experimentation

Many business people also don’t like experimentation when it comes to customer treatment decisions. They are unhappy with the idea that some customers will be subjected to an experimental approach that is likely to be worse than the default. They want to treat every customer the best they can and often have a hard time letting go of this approach. They too may resist experimentation.

From a business perspective, experimentation is a way to balance the short-term with the long-term. Experimenting with an alternative approach means risking treating some customers sub-optimally in the short term to build understanding about customer responses for the long term. Unless an organization experiments with how it treats its customers, it will not learn about their likely responses to alternative approaches, and it will be locked into a short-term, “what works now” mindset. Experimentation is going to be more and more important. Companies that can use their data to push the envelope—to get better and better—will outperform those that stick to the tried and true. Making test-and-learn part of your normal approach to business will be important, but not easy.

The importance of control groups

A control group is a group of customers (or partners or suppliers) who are set aside when a new approach is attempted. The control group continues to be treated following the old approach, while the remainder of the population is treated following the new approach. When first implementing a Digital Decisioning context, this means using the old approach to decision-making on some customers while treating others using Digital Decisioning. For instance, new customer retention using a Digital Decisioning system might be used to calculated retention offers for most customers, while a small number are still getting the default offer of the month. Over time, as new approaches are tried using the experimentation capabilities built into Digital Decisioning, a control group is formed by ensuring that the “old” approach continues to be used on some transactions—even if most are now processed using one of the new approaches.

Control groups are critical to effective analytic decision-making. Gary Loveman, the current CEO and President of Harrah’s Entertainment and an ex-Harvard Business School professor, is famous for his analytic approach to running the company and for the success of this approach. In particular, he has established a culture of experimentation and analytic decision-making. Loveman is often quoted as saying that there are three ways to get fired at Harrah’s: steal from the company; harass customers or other members of staff; or institute a program without a control group. This is a perhaps extreme point of view, but the importance of control groups should not be underestimated.

Building or buying a Digital Decisioning system, like any investment an organization makes, must show a return. It must be possible to show the positive results of the system in terms of reduced cost, increased revenue, or improved profit margins. Because Digital Decisioning typically enhances rather than replaces existing systems, the focus is generally on how it can improve the effectiveness of the organization as well as the more common focus on efficiency found when calculating the value of systems. To establish the increased effectiveness of the system, you need a baseline—that is what a control group gives you.

In the customer retention example above, if we identify a control group out and use our previous approach for them, we have a basis for comparison. We can route some percentage of customers who are calling to cancel to call center representatives who do not have the new system. This is the control group. After a while, we will have data that shows who was retained in the control group and who was retained in the group processed with the new system. Any improvement from the system will be clear and unambiguous. In contrast, with no control group, we would be comparing results now with results from before the system was implemented. Any difference might be explained by the system but might also be explained by changes in the way competitors price their plans, by new advertising campaigns, or by other factors outside the system. Proving the new system is effective would, therefore, be much harder.

Experimental Design

Experimentation and control groups are important in Digital Decisioning. It is not enough, however, just to run experiments. You need to design these experiments so that the data you collect will allow you to compare results and learn from the experiments. This requires effective experimental design. The classic work on experimental design is Ronald Fisher’s book The Design of Experiments, first published in 1935 (Fisher, 1971). In it, he outlines the key elements of experimental design. All of these are worth keeping in mind as you design your experiments:

  • Comparison: It must be possible to compare the results of your different experiments. You need a control group, so you can compare each experiment to the control group. You also need to make sure that the different experiments are similar enough to be compared. For instance, running two experiments with different customer retention offers will not result in useful data if the average value of offers in one experiment is twice as high as the value in the other. Think about the elements of an experiment that will allow the results to be compared effectively and used to convince someone that the comparison is valid.
  • Randomization: Which approach gets used should be determined randomly, as noted in “Build Test and Learn Infrastructure” in Chapter 6. Be careful this is not undermined by elements outside the system. For instance, you might experiment with a new approach by having three our of your 30 call center representatives use the new approach and the others the old one. If the assignment of calls to the representatives is geographic or by product type, then you may not get a random assignment.
    One of the most powerful features of Digital Decisioning when it comes to experimentation is the sheer number of transactions involved. Because you are handling thousands or tens of thousands of decisions, you have plenty of transactions that can be randomly assigned to experiments to create statistically significant result sets.
    Again, be careful with external factors. If you are handling self-selecting customers, such as those who responded to a survey, you may need to consider how this group relates to the whole population when assessing randomness.
  • Replication: Any experiment must be repeatable to be valid. This is not generally an issue with Digital Decisioning systems, as they are inherently repeatable: the same business rules are being applied to every transaction in an experimental group. The number of transactions involved also helps, as the transactions in an experimental group can be broken up into days, weeks, or months for comparison.
  • Blocking: Some elements of a person or transaction may be significant to an experiment. For instance, we may expect a different response to the same retention offers for customers who have never renewed (this is their first year) and those who have renewed before (second or subsequent year). Blocking is the use of this information to divide possible transactions or customers into blocks before randomizing them into an experiment. Rather than randomizing all customers calling in to cancel their service into our experiments, we might first divide them up into first-year customers and others before randomly assigning some of each category to an experiment.
  • Orthogonality: In any experiment, there are only a certain number of independent or “orthogonal” elements that can be considered. If two things are related—such as the length of time someone has been a customer and the total value of their business to date—then these need to be considered together in experiments. Understanding the “orthogonality” of your experiments will help determine what can and cannot be reasonably compared between them.
  • Factorial experiments: Sometimes there are a number of factors you wish to consider. You could test each one sequentially by designing an experiment to test the various options for the first factor, gathering results, then moving on to the second factor. Factorial experiments take multiple factors and combine them into the minimum number of experiments. For example, if you want to experiment with the value of a retention offer and the length of renewal contract necessary to get the offer, then you have two factors to consider. If you decide there are three levels of value (high, medium, and low) and two renewal periods (one year and two year), then a factorial experiment would consider every possible combination plus a control group—seven experiments, as shown the table below.
Experiment Value Renewal
1 (control group)
2 Low 1 year
3 Low 2 year
4 Medium 1 year
5 Medium 2 year
6 High 1 year
7 High 2 year