Slides used for the "A/B testing ultimate guideline" presentation in GPec Romania, May 2018. In these slides you can find a guide about how to design, develop and analyze a/b testing and personalization in digital business such e-commerce, lead generators, etc.
5. 5A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
In web analytics, A/B testing (bucket tests or split-run testing) is a controlled
experiment with two variants, A and B.[1] [2] It is a form of statistical hypothesis
testing or "two-sample hypothesis testing" as used in the field of statistics. A/B testing
is a way to compare two versions of a single variable typically by testing a subject's
response to variable A against variable B, and determining which of the two
variables is more effective.
Wikipedia
13. 13A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
¿Why A/B testing
(and personalization)
is so needful in a
digital business?
14. 14A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
A/B testing is the main tool we have to prove that something we have built a
hypothesis around (action, strategy, change) will improve our results in our digital
business, starting from data.
A/B testing allow us to validate with quantitative data how useful and needful a
changue could be in our context in a specific moment.
It helps us to avoid guessing or self-reference design. Both things are great problems
in digital design processes today.
It helps us in our constant learning about how can we built a stronger and more
efficient digital product.
28. 28A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Goals Data Hypothesis
Facturación
GA / Digital
Analytics
Brainstorming
Leads
Datos
cualitativos
5 why rule
Incrementos % Logs
Covariation
hypothesis
Engagement CRM / LTV
Correlation /
Cause-effect
User Research FUDs
29. 29A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
A hypothesis should be a statement, a proposition that you make to describe what
will happen in a system given specific circumstances when we modify some
variables. It often follow this form: “If we do X, user will do Y with will impact metric
A”.
Hypotheses should be descriptive but short and they have to generate a
consequence in the system.
Example: “If we improve our information architecture user will find easyly our
products, so our bounce rat will decrease and tire will be a bigger amount of users
starting checkout process”
36. 36A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Goals Data Hypothesis Solutions Test
Facturación
GA / Digital
Analytics
Brainstorming UX A/B test
Leads
Datos
cualitativos
5 why rule Traffic MVT
Incrementos % Logs
Covariation
hypothesis
Technology Split
Engagement CRM / LTV
Correlation /
Cause-effect
Business model JS Custom
User Research FUDs Triggers Personalization
37. 37A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Testing - scientist method:
Independent variable: The variable you´re going to study and modify, and the one
that affect results in the test. You should control that variable. Classic example:
nutrition. Digital Example: information architecture.
Dependent variable/s: DIt depends on the independent variable and its value is
affected by it. They respond to the change made to the independent variable. Classic
example: nutrition. Digital example: bounce rate.
Controlled variables: Controlled variables are quantities that should remain
constant to make the test reliable, and they has to be observed them carefully
because any change in these can modify test results. Classic example: training.
Digital example: Release of new Adwords campaing.
39. 39A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
How many users do I need for a solid testing? - https://www.optimizely.com/sample-
size-calculator/:
40. 40A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
How many users do I need for a solid testing? - https://www.optimizely.com/sample-
size-calculator/:
41. 41A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
How many users do I need for a solid testing? - https://www.optimizely.com/sample-
size-calculator/:
42. 42A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Conversion rate metric we´re measuring:
conversion rate, download, bounce rate, etc
Miminum relative change we want to detect
in that metric. In this case we want to detect
a range between 0,75%-2,25%
How reliable your test is. Tthe probability
that the results don´t come from chance /
luck (p-value)
How many users do I need for a solid testing? - https://www.optimizely.com/sample-
size-calculator/:
43. 43A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Traffic volume - statistical significance:
Can I do A/B testing if I don´t have enough amount of traffic to gain statistical
significance?, Should I do it?, is A/B testing only for big projects?
44. 44A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Testing type:
A/B Test - Split testing: The metrics tested of two versions of a page — version A
and version B — are compared to one another. Site visitors are bucketed into one
version or the other. There´s only one variable changing at the same time.
MVT Test: Compares a higher number of variables at the time times. Several items
(variables) are tested in the same main layout.
Personalization: Based on cookies and navigation paths, we offer a customized
digital experience to the users that fix into a set of rules previously defined.
50. 50A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Segments and distribution:
Before launching any kind of A/B testing we have to set the amount of traffic we
want to send to the test (statistical significance - reliable) and we have to define the
right segments tu run the test.
51. 51A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Segments and distribution:
Main segments we have to consider:
Desktop / Smartphone: Una Same solution can be solved so differently depending
on the interface type, so we can have very different results from the same test.
Countries: Same experiment can give us different data depending on the country we
´re runing it.
Business units: Although we´re working in the same e-commerce, each one has
several business units or activities. Not all actions will give us same feedback
depending on the business unit.
Traffic channels: Different results depending n the origin of the traffic.
52. 52A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
Segments and distribution:
Main segments we have to consider:
Date: Different results depending on running the test in Christmas, summer, etc.
Political / Society variables
Login or not
Weather