One of the most commonly asked questions is “when is an MVT experiment or AB test finished?” Is it at 30 days...? 100 conversions...? 10,000 visitors...? The short answer is... it depends.

1. The Finishing Line- when is a
Multivariate Test finished?
By Joe Doveton
Oban Digital

2. When Is A Test Finished?
• One of the most commonly asked questions is “when
is an MVT experiment or AB test finished?”
• Is it at 30 days...? 100 conversions...? 10,000
visitors...?
• The short answer is... it depends

3. Known and Unknown Variables
A test is made up of known and unknown
variables...

4. Things you know:
– Likely visitors to
test page(s)
– Likely conversion
rate for the
original (control)
–Numbers of design
variations in the
test
Things you don’t:
– The improvement
(or deterioration)
in the conversion
rate generated by
the challenger
designs

5. Understanding Test Data
Conversion rate for the original is 51.68%
The leading challenger is Combination 7
with an 80% probability of beating
the original
This is a 2.40% uplift
in Conversion Rate vs
the original

6. Margin of error
is important because it effectively
tells you if you have a big enough
sample. In an ideal world one
would wait until the margin of error
was 0%

7. Margin of error
• In this test there is 0.7% margin of error in
the sample
• This means that the original is converting at
51.68% plus or minus 0.7%
• The best case scenario for the original is
therefore to add 0.7% to the conversion rate
making 52.38%
• The worst case scenario for the test leader is
to subtract 0.7% from that conversion rate,
making a CR of 52.22%
• This means the test is not quite statistically
valid-we could run the test a little longer to
get a big enough sample
• Despite this, there is enough evidence to
apply Combination 7 as a test winner- 80%
confidence and 2.40% uplift over the original
plus £10k in increased revenue

8. In some tests it is not practical to
wait until you have 0% margin of
error, especially on low traffic
pages.

9. Sometimes you just have to make a
call. Here are some tips to help you
decide when your test is finished.

10. Fit test type to visitor profile
• With low traffic pages
make small changes and
test via AB testing. You
will get a quicker result
and you can iterate
• With higher traffic pages
you can use MVT. MVT
has the advantage of
allowing you to test many
variations at once but still
isolate the impact of
individual on-page
elements

11. Brainstorm
• A test is only as good as the
ideas that go into it
• Manually go through funnels
• Use analytics
• Analyse competitors
• Group and ungroup elements
• Prioritise tests- do the low
hanging fruit first
• Set yourself cut off points

12. Test goals
Match your test goals as closely as
possible with your regular goals set up
in analytics
Test goals should be consistent with the
elements that you want to change- for
example if you want to change CTR
changing the footer navigation isn’t
likely to contribute much to this

13. Keep Things Manageable
• If using MVT keep your test down
to a manageable number of
combinations
• Test only a few elements
together concurrently
• Each time you add in a new
element, you double the length
of time it takes to get a
statistically valid result (same
with segments)
• No-one wants to read a test with
128 possible test combinations
(sorry Mr. Taguchi)

14. Check The “Elements” View
• With MVT, one can check both
the interaction between
elements (the ‘combination’)
and the individual elements in
isolation
• Often, the elements view will
give you a different view of the
data
• In a test with no clear winners,
there may be evidence of
specific elements driving
conversions that allows you to
plan a follow-on experiment

15. Confidence Level
• 100% confidence results in
test are rare. Very rare- even
in AB tests.
• Set yourself an acceptable
confidence level and go with it
• Anything over 70% is pretty
good- 80% and above is
exceptional
• PS this also works in reverse
with losing tests
• Be prepared to call winners
and losers
• Accept that inconclusive
results are part of the game

16. Be Prepared To Iterate
• So your first test result was
inconclusive. Great! Why
great? It gives you an
opportunity to re-evaluate the
design
• You start to gain insight into
what elements and what
changes affect user behaviour
• Inconclusive test results give
you the chance to try
something else
• Each test is a stepping stone-iterate,
test and move on

17. Lifetime Value of Test Result
• A small uplift over
a two week test
period might leave
you feeling
underwhelmed
• However, what is
that uplift worth
when extrapolated
over a whole year?
• Even a modest test
result could transform
your business

18. There’s No Such Thing As Bad Data
• Everybody loses tests
• It’s part of the game
• Losing tests stop you
from going off with
design changes in the
wrong direction
• Losing tests help to
point you back in the
right direction

19. Follow The Money
• ... and finally... follow the
money
• Many platforms allow you to
track revenue as well as
conversions- if not, integrate
with Google Analytics
• Even if you increase
conversions it may not
increase revenue- your design
could make it easier for
customers to buy your lower
priced items
• Track the test impact on your
bottom line either through
your test tool or your analytics

20. Contact Oban Digital
Phone: +44 1273 936095
Email: sales@obandigital.com
Website: www.obandigital.com
Twitter: twitter.com/ObanDigital