How automation, AI, and dashboarding can help scale experimentation programs
If you are running 10+ experiments monthly across multiple teams, you know how important efficiency is. Moreover, available resources may not scale uniformly, making automation increasingly necessary to streamline processes. The challenge intensifies with execution quality when coordinating programs across diverse teams.
While having clear standards and disciplined workflows is beneficial, integrating automation, leveraging AI, and utilizing dashboards significantly enhances any experimentation program. This not only addresses scalability issues but also provides stakeholders with more time to focus on the core aspects of their work.
This webinar will be your script to plan and build automation, leverage AI and effective dashboarding to scale experimentation programs. Sign up now!
12. 12
“If you don’t have a process, you don’t know what you’re doing”
E. Deming
13. 13
Step 1: Start with a structured process
1) Ad-hoc experimentation 2) Dedicated experimentation 3) Advanced experimentation 4) Embedded experimentation
Maturity state
Single touchpoint Multiple touchpoints Full customer journey All (marketing & product related)
Experimentation scope
1 person Small team(s) Multiple organized teams Embedded (no CXO team)
Teams
Basic Structured process Advanced / automated Democratized
Process
1: Start structuring the
program
14. 14
Step 1: Start with a structured process
Ensure a solid workflow
15. 15
Step 1: Start with a structured process
Templates Subtasks Sprint management
16. 16
Step 2: Start automizing standardized
tasks
1) Ad-hoc experimentation 2) Dedicated experimentation 3) Advanced experimentation 4) Embedded experimentation
Maturity state
Single touchpoint Multiple touchpoints Full customer journey All (marketing & product related)
Experimentation scope
1 person Small team(s) Multiple organized teams Embedded (no CXO team)
Teams
Basic Structured process Advanced / automated Democratized
Process
2: Take your first
automation steps
17. 17
Automated Program Dashboarding:
Filter out all results
and experiment data
Quickly find all
your reports
based on
advanced
filters
View results from whole
program or specific team(s)
Succesdata
Compare
success ratios
to understand
what works
and what not
18. AI to increase efficiency
Generate quick ideas or substantiate
existing ideas with generative
content or create your own
database for trend & meta-analyses
1. AI Ideation
Generative AI can help create
assumptions on how to interpret
data and create hypotheses.
2. AI enriched
opportunities
AI can help with inspiration and quickly
generate multiple versions (viewports /
devices) of existing designs
3. Faster design
AI can help to create assumptions on
how the data can be interpreted in user
behavior and attitude to generate follow
up ideas.
4. Rich validation
21. 21
Step 3: Automation by default
1) Ad-hoc experimentation 2) Dedicated experimentation 3) Advanced experimentation 4) Embedded experimentation
Maturity state
Single touchpoint Multiple touchpoints Full customer journey All (marketing & product related)
Experimentation scope
1 person Small team(s) Multiple organized teams Embedded (no CXO team)
Teams
Basic Structured process Advanced / automated Democratized
Process
3: Automation at the
core of the program
22. 22
Experiment data management & monitoring:
Basic decision
support on
what to do.
Only show
the
experiments
you need
Splitwatch+ shows all
experiments and its relevant
data at one glance
See all
running
experiments
and select KPI
to call
23. 23
Experiment database:
Summary of results +
option to download
full report
Build custom filtering
& dashboarding for
specific research topics
Multiple variables available
to easily search through it
A database
with all
experiments
24. 24
Experiment database:
Job
Completion
Surveys Funnel
analysis
Return
rate
analysis
Customer
Journey
Mapping
A/B test
analysis
Landing
page traffic
analysis
Refined
opportunity
Moderated
usability
study
Expert
review
Competitor
scan
Prototype
test results
Funnel
analysis
Exit
survey
results
Preference
test results
1: One place for the
entire organisation to
log insights
2. A uniform way to
add new insights using
a clearly defined data
model & forms
3: The ability to easily find
and combine insights.
27. 27
CXO Ecosystem:
Research
inspiration
Better predictions
due to prior data
& insights
Design
inspiration
Automated Program
Output overview
Splitwatch+
automated data &
monitoring tool
Automated
reporting
Structured
Opportunity’s,
Areas, Solutions
and their
relations (OST)
Faster “meta-
analyses”
Experiment repository stores all
important experiment data
28. 28
Summary
1) Ad-hoc experimentation 2) Dedicated experimentation 3) Advanced experimentation 4) Embedded experimentation
Maturity state
Single touchpoint Multiple touchpoints Full customer journey All (marketing & product related)
Experimentation scope
1 person Small team(s) Multiple organized teams Embedded (no CXO team)
Teams
Basic Structured process Advanced / automated Democratized
Process
1: Start structuring the
program
29. 29
Summary
1) Ad-hoc experimentation 2) Dedicated experimentation 3) Advanced experimentation 4) Embedded experimentation
Maturity state
Single touchpoint Multiple touchpoints Full customer journey All (marketing & product related)
Experimentation scope
1 person Small team(s) Multiple organized teams Embedded (no CXO team)
Teams
Basic Structured process Advanced / automated Democratized
Process
2: Take your first
automation steps
30. 30
Summary
1) Ad-hoc experimentation 2) Dedicated experimentation 3) Advanced experimentation 4) Embedded experimentation
Maturity state
Single touchpoint Multiple touchpoints Full customer journey All (marketing & product related)
Experimentation scope
1 person Small team(s) Multiple organized teams Embedded (no CXO team)
Teams
Basic Structured process Advanced / automated Democratized
Process
3: Automation at the
core of the program
31. 31
Benefits of automation in experimentation
√ Scaling will become easier and less resource heavy
√ Less quality loss due to automated data-prep and decision support
√ Better, clearer overview of all CXO output, results, impact and ROI
√ Save time for deeper analyses, designs, interpretations and follow ups.
√ More fun! As repetitive tasks are mostly automated.
But
➢ Beware of rubbish in – rubbish out: Prioritize clear process standards
➢ Standardisation v.s. customization
➢ AI isn’t trustworthy (yet)