The Digital Transformation Playbook: Rethink Your Business for the Digital Age

1. Define the question and its variables
Convergent Experimental Method
Question statement
Independent & dependent variables
4. Validate your sample
Unit of analysis
Signal-to-noise
n = ?
2. Pick your testers
3. Randomize your test and control
5. Test and analyze
6. Decide
7. Share learning
Figure 5.2
The Convergent Experimental Method.

---

144
I N N O V A T E B Y R A P I D E X P E R I M E N T A T I O N
r
 Dependent variable (or effect): This is the factor that you expect may 
be influenced by your new innovation. It is a measure of the impact of 
what you are changing.
Step 2: Pick Your Testers
The next step is to select who will conduct the experiment. This could be the 
managers who have developed the possible innovation or an impartial party.
Because it follows formal experimental practices, the test will require 
some statistical knowledge or tools. Many tests can be automated with soft-
ware tools. Services like Optimizely provide self-service tools to run A/B tests 
on webpages’ content or design. E-mail service providers like MailChimp 
include tools for running A/B tests on e-mail content or subject lines. (These 
services are inexpensive or even free for small businesses.) Your employees 
can be easily trained to run and record these kinds of experiments.
However, for more complex phenomena, such as competing retail 
designs, testing will be less automated, and more statistical knowledge is 
required. For this reason, an organization may want to designate a testing 
team to run valid experiments for innovation projects. Such an internal 
team can be called on to ensure the experiment is set up properly and to 
assist in analyzing the data afterward.
Step 3: Randomize Your Test and Control
Before running a convergent experiment, you must identify a population 
whose responses you want to test (frequently your customers or a particu-
lar subset of your customers).
Next you randomly assign members of that population to one of two 
groups:
r
 The test group (or treatment group), which receives the experience or 
offer you are testing
r
 The control group, which does not
Randomizing the test and control groups is the step where most mis-
takes happen in convergent experiments. A business will identify its ques-
tion and then carefully choose who will go into the test group versus the 

---

I N N O V A T E B Y R A P I D E X P E R I M E N T A T I O N

145
control group. When it first ran experiments on retail innovations in its 
stores, Petco made this mistake consistently. Seeking to test innovations in 
the “optimal” conditions, the firm would roll them out in its thirty high-
est-performing stores nationwide. It would then compare results from this 
group and results from its thirty lowest-performing stores. Not surprisingly, 
innovations that tested as “beneficial” among the superstar group would 
sometimes disappoint when rolled out nationally across all locations. Petco 
has since learned to avoid this mistake.
28
Step 4: Validate Your Sample
Next you need to make sure you have a valid sample size. That starts with 
identifying your unit of analysis. For example, if you are testing an offer 
sent to individuals in your database, then the unit of analysis is the indi-
vidual respondent. But if you are testing two versions of a retail store layout, 
then the unit of analysis is the store. (You are only able to compare the 
effects of one store to those of another.)
Once you know your unit of analysis, your sample size is simply the 
number of units that you place in each of your test and control groups. For 
example, if you have 600 e-mail addresses and you send three versions of 
an e-mail, each to 200 recipients, then your sample size is n = 200.
What is a statistically valid sample size? The typical rule of thumb is to 
have n = 100, at a minimum, in each group you are comparing. However, 
depending on your signal-to-noise ratio, you may need a larger sample size. 
If the impact of your innovation is large, you may be able to measure it with 
a sample of n = 100. But if the impact is much more subtle (e.g., a small 
lift in customer conversion rate), you will need a larger sample so that the 
effect of your treatment is greater than the margin of error. (A larger sample 
yields a smaller margin of error.)
Step 5: Test and Analyze
Now you are ready to run your test. The team conducting your experi-
ment will gather data over a predetermined time span. Then they will need 
to analyze the data to see whether there are differences in the dependent 
variables you are measuring and, if there are, whether those differences are 
statistically significant.

---

146
I N N O V A T E B Y R A P I D E X P E R I M E N T A T I O N
When you do measure and analyze the results, it is important to gather 
data beyond the dependent variables that you chose in step 1 to define suc-
cess for your experiment. Even if you have a clear answer (“yes” or “no”), 
you will also want to know why. When the Family Dollar discount store 
chain tested a plan to add a new section with refrigerated foods, it mea-
sured whether customers bought enough of the cold foods to justify the 
cost. The test said yes. But the chain also found that customers purchased 
more dried goods after stores introduced the refrigerated section; the result 
was a much greater boost to profitability.
29
Step 6: Decide
After analyzing the results of your convergent experiment, it is time to 
make a decision based on the findings. This is where having agreed on 
your definition of success in step 1 will pay off.
If you do find a desired improvement from your innovation test, the 
story may not be over. This should often lead to further iteration and test-
ing of additional ideas to see if they can lead to greater improvement. In 
the 2008 presidential contest, the Obama campaign ran test after test, 
examining the effects on fund-raising appeals of changing many different 
elements—the subject of the request, the kind of photos and videos, the 
“call to action” words on the button that led you to a donation page. Each 
subsequent test added a bit more learning, but the cumulative effect was 
to raise the final rate of conversions—from e-mail to website to volunteer 
sign-up to donation—by 40 percent, or an estimated $57 million of addi-
tional fund-raising.
30
Step 7: Share Learning
Once you complete your analysis, it is essential to capture and share the 
learning of your experiment. If you are doing a battery of experiments on 
the same variables, this process can happen at the end rather than after each 
step. But it is critical to both document what you learned and communicate 
your findings to others in your organization who could benefit (and could 
avoid any of the same mistakes).
You can find a list of sample questions to use in capturing and sharing 
learning from any convergent experiment with your team in the Tools sec-
tion of http://www.davidrogers.biz.

---

I N N O V A T E B Y R A P I D E X P E R I M E N T A T I O N

147
Tool: The Divergent Experimental Method
The second tool is a guide for running divergent experiments. This method 
is particularly useful for innovations that are less defined from the out-
set, such as new products, services, and business processes for your orga-
nization. Innovation projects using divergent experimentation tend to be 
highly iterative and may span weeks or months.
You can see the ten-step Divergent Experimental Method in figure 5.3. 
Its steps fall into three stages: preparation, iteration (steps that repeat sev-
eral times), and action.