1. A/B Testing for Apps
How I Learned to Stop Worrying and Love
Data-Driven Decisions
Prepared for:
Coca-Cola Mobile Innovation Workshop
September 24-26, 2013
Shanghai, China
A/B Testing for Apps

2. User Base
Publish two different versions
of your app...
50% sees version A
...and see which one is driving
desirable user behavior.
50% sees version B

3. Which Version Won?
Version A
Version B
Source: WhichTestWon

4. Version B: 114% Improvement
✗
Version A
Version B
Marketer’s Surprise: ‘FREE’ Loses
✔

5. Multivariate Testing
• Every screen has X components (ex: Marilyn’s hair)
• For each, we can test Y variations (ex.: Green)
• In total, we have [Y1 x Y2 x Y3] combinations

6. How Obama Raised $60 Million
Source: Optimizely

7. Four Button Variations

8. Six Media Variations

9. 24 Combinations!

10. And the Winner is...
+40%
increase in conversion
rate
2.9 million
additional donators
$60 million
value of additional
donations

11. Obamalytics
• Original Conversion Rate:
• New Conversion Rate:
8.3%
11.6%
• 10 million signups from New Version would have been
7.12 million signups with the Original Version
• +2.88 million additional signups
• $21 average donation per signup
• Approximately $60 million in additional donations

12. Testing to
Increase Virality
Headline
Call to Action
Workflow

13. Data-Driven App Marketing
• App marketing and analytics is in early stages
• App stores are opaque, approval processes slow down
iteration cycles
•
Few options for making decisions
• HiPPO: Highest Paid Person’s Opinion
• IHOP: In-House Optimization
• Scalable, reliable app measurement tools are emerging

14. Thank You!
www.splitforce.com
For more information:
Zac Aghion, CEO & Co-Founder
zac@splitforce.com
China: (+86)1592-1631-924
USA: (+1)617-750-6684