Alexander discusses the approach to hypothesis driven development at APPRL and how you can develop your product in the right direction.

1. JUNE 27TH 2019 — ALEXANDER BERTHOLDS, HEAD OF DATA AT APPRL
Hypothesis driven
development
UXDX Stockholm

2. Develop your product in
the right direction
Hypothesis driven development is a scientific
method adapted and used in tech
Lab Rats used for A/B testing in medicine
Process
• Analyze your users and create a hypothesis of
what feature they need
• Test the feature on a big enough sample of your
users (A/B test)
• Learn from the test you ran - was the hypothesis
correct or not?

3. Define your hypothesis
?
• Based on the insight …
• We think …
• Will make …
• So that …

4. Case:
New signup page
• Based on the insight that new users abort
the signup page to return at a later time
• We think that saving the user’s provided
information
• Will make the signup flow less tedious when
re-typing info isn’t necessary
• So that the conversion on the signup page
will increase with at least 4 percentage
points
?

5. Where do we find the
data to get good insights? User research
Product usage
data
Reports from
customer
support and
sales
Experience
A.k.a gut feeling

6. Creating an A/B test
• Split your users in two groups,
randomly, and expose each group
to one of the two features
• Compare the chosen evaluation
metric between the two groups
A/B TEST
RESULT
No saved
information
52% conversion
Save
information
54% conversion
• How sure can you be that the
reality is reflected in the result?

7. The statistics behind
experiments
When you run your A/B test your
result will be a random sample from
a normal distribution with the mean
at the “true” performance of your
features
The spread of conversion rates when running
1000 repeated experiments on the same
features
In reality the true mean is never known!

8. The statistics behind
experiments
This means that even if feature B is
better than feature A you will
sometimes get another result.
The frequency of experiments that
produces “false” results is determined
by the selected significance level and
power

9. Significance level
The rate of experiments that will falsely tell you
there is a difference between A and B even
though there is no real difference
Typically 5% is used
Power
The rate of experiments that will correctly tell
you when there is a difference between A and
B
Typically 80% is used
Determined by sample size and minimal detectable effect

10. What is minimal
detectable effect?
• The MDE is the smallest change you
will be able to detect with the chosen
power
• If you think conversion rate will go at
least from 50% to 54% then the MDE is
4 percentage points
• Any changes smaller than the MDE will
be considered insignificant

11. Why not use a really small
minimal detectable effect?

12. The balance act of choosing
experiment parameters
Decreasing the minimal detectable effect gives
either higher significance and lower power
levels or requires bigger sample size
Increasing the sample size will either give you
longer test time or it will not be possible with
the current user base
Not keen on maths? Check out
http://www.evanmiller.org/ab-
testing/sample-size.html

13. Choosing evaluation
metric
• High level metrics like revenue that are very
relevant to your overall goals but often move
slowly
• Low level metrics like conversion rate that
are proxies of the overall goal but can be
evaluated after a short period of time

14. Develop your product in
the right direction
Hypothesis driven development is a scientific
method adapted and used in tech
Lab Rats used for A/B testing in medicine
Process
• Analyze your users and create a hypothesis of
what feature they need
• Test the feature on a big enough sample of your
users (A/B test)
• Learn from the test you ran - was the hypothesis
correct or not? What does that say about your
users?

15. Reach out!
alexander@apprl.com
alexander.bertholds@gmail.com