The Digital Transformation Playbook: Rethink Your Business for the Digital Age

Divergent experiments are best suited for learning that explores options, 
generates insights, asks multiple questions at the same time, and, when 
done right, generates new questions to explore in the next iterative stage. 
(See table 5.2.)
Both types of experiments increase our knowledge and test our 
assumptions. Both involve looking outside the organization for answers, 
and both require willingness to learn versus just planning and deciding. 
But the approach of each type is quite different. Let’s look at them in detail.
Table 5.2 
Two Types of Experiments
Convergent Experiments
Divergent Experiments
Example: A/B feature testing or a pricing test
Example: putting a prototype in the hands
of customers
Formal (scientific) experimental design
Informal experimental design
Asks a precise question or finite set of 
questions
Poses an unknown set of questions
Seeks to provide an answer
May provide an answer or raise more 
questions
Needs a representative customer sample
(test and control groups)
Needs the right customers (who might not 
be average customers)
Needs a statistically valid sample
Sample size may vary
Focused on direct causality
Focused on gestalt effects and meaning
Goal is to test the thing itself
Goal is to test as rough a prototype as 
possible for the question ( “good enough”)
Confirmatory
Exploratory
Useful for optimization
Useful for idea generation
Common in late stages of an innovation
Common in early stages of an innovation
IN COMMON
Increases knowledge
Tests assumptions
Looks outside for answers
Requires willingness to learn versus decide

---

130
I N N O V A T E B Y R A P I D E X P E R I M E N T A T I O N
Convergent Experiments
The key element of every convergent experiment is the initial causal 
hypothesis: “If I add this feature, customers will spend more time on my 
site.” Or “If I change this interaction, customers will spend more money 
in my store.” Convergent experiments are critical for cases where it is not 
enough to know the correlation between two events; you also need to verify 
which event is causing the other.
Convergent experimentation is applicable in a variety of contexts. It 
can be used with any digital product or service (website, mobile app, soft-
ware, etc.) to test and improve any element of the customer experience. 
This is why not only Google but also every major Internet service, such as 
Amazon or Facebook, is constantly running A/B tests, in which two sets of 
customers see the same webpage (or the same e-mail) with one difference 
in design and the company measures any difference in customer behavior 
or response. Facebook is famous for experimenting with the News Feed of 
its users to find the right balance of photos versus text posts versus videos, 
the friends a user is more interested in hearing from, and the kind of con-
tent that is interesting only in the short term or meaningful to a friend who 
only logs in to Facebook several days later.
However, convergent experimentation can be applied in nondigital 
environments as well. These kinds of experiments are at the heart of data-
driven strategies to optimize the guest experience and loyalty rewards given 
to customers of hotels, airlines, and resorts. When convenience store chain 
Wawa is planning changes to the food menu, it will run experiments to 
measure not just if customers buy the new item but also if there’s an impact 
on the overall profitability of customer visits.
7
Convergent experimentation is often used in communications and 
direct marketing. In both presidential campaigns of Barack Obama, contin-
uous, rapid experiments on e-mail subject lines and website page designs 
helped to dramatically increase their effectiveness in signing up new sup-
porters and garnering more donation dollars. Starting in the pre-Internet 
era, Capital One bank used convergent experiments to test the right pro-
motional offer, the right target audience, and even the right color of enve-
lopes as it mailed out credit card invitations. By running tens of thousands 
of experiments each year that focused on customer acquisition and lifetime 
value, it grew from a small division of another bank into an independent 
company worth $42 billion.
8

---

I N N O V A T E B Y R A P I D E X P E R I M E N T A T I O N

131
A convergent experiment can be as expensive as testing two different 
store layouts for a retail chain or as cheap as sending two versions of an 
e-mail promotion, each to a different group of randomly selected custom-
ers, and comparing the responses.
Because convergent experimentation needs to measure causality, it 
needs to adhere to the key principles of formal scientific experiments:
r
 Causal hypothesis—so that you have an independent variable (the cause) 
and one or more dependent variables (the effect)
r
 Test and control groups—so that you can see the difference between 
those who are exposed to your stimulus and those who aren’t
r
 Randomly assigned participants—so that an external factor doesn’t 
influence the outcome of your test group
r
 Statistically valid sample size—so that the differences you measure can 
rise above the noise of random fluctuations
r
 Blind testing—so that you avoid the Hawthorne effect, where those 
involved in the experiment unintentionally influence its outcome
Common mistakes in convergent testing mostly center on improperly 
assigning participants to the test and control groups. For example, a retailer 
might select a set of participants (its top customers or its better-performing 
stores) for a new treatment and erroneously assume that “everyone else” 
(all its other customers or stores) can serve as an equivalent control group.
Some of the key writers on convergent experimentation for business 
include Stefan Thomke, Thomas Manzi, Eric T. Anderson, and Duncan 
Simester.
9