SlideShare ist ein Scribd-Unternehmen logo
1 von 25
Downloaden Sie, um offline zu lesen
10 Tactics for Building an Optimization Culture
OPTIMIZELY & BROOKS BELL
3.4.2014
Welcome!
Gregory Ng
Chief Marketing Officer
Brooks Bell
@GregoryNg
Kyle Rush
Head of Optimization
Optimizely
@kylerush
10 Tactics for Developing a Successful
Experimentation Culture
What we’ll cover ...
1. Best practices from top testing
organizations
2. The people behind testing
3. How to organize your program
4. How to keep the process
manageable
5. How to include others and build
excitement
Successful experimentation culture
must be data-driven.
What does it mean to
be data-driven?
Progress is compelled by data, rather than by intuition or
personal experience.
Evidence-based decision making.
Using data to inform key decisions.
How data-driven are you, actually?
•

Do you dictate strategy without referencing data?

•

Do you rely on the person with the most experience to
make the ultimate call?

•

Do you use words like “think” and “feel” when talking
about strategy?

•

Does your team trust the data?
How do you establish a successful
testing culture?
10 Tactics
1. Clear organizational structure
2. Identify the right talent
3. Establish stakeholders
4. Solicit test ideas from everyone
5. Prioritize tests
6. Share results and successes
7. Build a knowledge base
8. Don’t rush into MVT tests
9. Balance learning and “wins”
10.Go beyond CRO
#1 – Choose the right
organization structure
• No “one-size-fits-all”
• One or multi-person
team
• Centralized or
decentralized?

Members of the Obama for America Team
#1 – Choose the right
organization structure
• Army of One

• Ringleader

• Fellowship

• Kingdom
#2 – Pick the Right Talent
• Qualitative and quantitative
combination
• Curiosity, creativity
• People skills
• Analytical mindset

VP of Marketing

E-Commerce
Director

Program Manager,
Retail
Optimizely’s customers come from a
variety of backgrounds

Product Manager
#3 – Establish Stakeholders Early On
• Identify the correct
stakeholders
• Secure an executive sponsor
• Develop an approval process
for tests
• Communicate and share
successes
#4 – Solicit Test Ideas from
Everyone
• Make it easy to submit
ideas
• Capture interest after
sharing results
• Follow up for every idea
• Create a scoreboard
• Attribute experiments to
team members

Example: Optimizely Scorecard
Vote on predicted winning variations
#5 - Know how to Prioritize Tests
ESTABLISH A SCORECARD
•Relevance to business goals
(KPIs)
•Potential impact
•Potential for learning
•Level of effort
•Time to significance and
available traffic
•Likelihood of implementation
#6 – Share Results and
Successes
•
•
•
•
•

Make it easy to understand
Clarify the primary goal
Share 100 percent of
results, regardless of
outcome
Standing meetings
Keep score

Presenting A/B test findings at company all-hands
#7 – Build a Knowledge Base
• Gain broader insights
• Lay a foundation for growth
• Eliminate single point of failure
• Increase accountability
• Standardize processes and
documentation
#8 – Don’t Rush Into MVT Tests
• Ideal for fine-tuning
optimization
• More complicated but less
effective
• Fewer learnings
#9 – Balance Learning and “Wins”
•
•
•
•

Always start with a hypothesis
Never settle for a flat result
Invitation for deep analysis
Segment results, target
accordingly
#10 – Go Beyond CRO
Optimization doesn’t need to occur in isolation.
Almost anyone can test.
For instance ...
#10 – Go Beyond CRO
• Designers

Variation A

Variation B

Walk designers through A/B tests so they can inform their designs with data
#10 – Go Beyond CRO
• Engineers

Launch of OFA 2012 New Donation API
A/B testing the new platform revealed an
architectural bug that was missed in QA
#10 – Go Beyond CRO
• Email and Content Marketers

A/B Tested Emails and Landing Pages for Subscribers

#dataculture
1. Clear organizational structure
2. Identify the right talent
3. Establish stakeholders
4. Solicit test ideas from everyone

In summary ...

5. Prioritize tests
6. Share results and successes
7. Build a knowledge base
8. Don’t rush into MVT tests
9. Balance learning and “wins”
10.Go beyond CRO
Thank you for reading!

Weitere ähnliche Inhalte

Was ist angesagt?

Prioritise: Kanban Recipe for Success Step 5
Prioritise: Kanban Recipe for Success Step 5Prioritise: Kanban Recipe for Success Step 5
Prioritise: Kanban Recipe for Success Step 5Jason Yip
 
Building Better Products, June 2015
Building Better Products, June 2015Building Better Products, June 2015
Building Better Products, June 2015Jason Fraser
 
Building a Culture of Product Experimentation by Optimizely PM
Building a Culture of Product Experimentation by Optimizely PMBuilding a Culture of Product Experimentation by Optimizely PM
Building a Culture of Product Experimentation by Optimizely PMProduct School
 
What Are the Product & Design Principles by FindMyPast PM
What Are the Product & Design Principles by FindMyPast PMWhat Are the Product & Design Principles by FindMyPast PM
What Are the Product & Design Principles by FindMyPast PMProduct School
 
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...CXL
 
Test Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew EakinTest Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew EakinQA or the Highway
 
The argument for agile development
The argument for agile developmentThe argument for agile development
The argument for agile developmentToby Osbourn
 
Changing business of testing - Testing Assembly Helsinki 2014
Changing business of testing - Testing Assembly Helsinki 2014Changing business of testing - Testing Assembly Helsinki 2014
Changing business of testing - Testing Assembly Helsinki 2014Vasco Duarte
 
Why does agile need unit testing
Why does agile need unit testingWhy does agile need unit testing
Why does agile need unit testingMohsen Bazmi
 
A/B Testing and the Infinite Monkey Theory
A/B Testing and the Infinite Monkey TheoryA/B Testing and the Infinite Monkey Theory
A/B Testing and the Infinite Monkey TheoryUseItBetter
 
Why unvalidated assumption is the enemy of good product
Why unvalidated assumption is the enemy of good productWhy unvalidated assumption is the enemy of good product
Why unvalidated assumption is the enemy of good productSeb Agertoft
 
Optimizely Demo Deck
Optimizely Demo DeckOptimizely Demo Deck
Optimizely Demo DeckMattharth
 
Mock Interviewing with Salesforce's Product Manager
Mock Interviewing with Salesforce's Product ManagerMock Interviewing with Salesforce's Product Manager
Mock Interviewing with Salesforce's Product ManagerProduct School
 
Test Estimation Hacks: Tips, Tricks and Tools Webinar
Test Estimation Hacks: Tips, Tricks and Tools WebinarTest Estimation Hacks: Tips, Tricks and Tools Webinar
Test Estimation Hacks: Tips, Tricks and Tools WebinarQASymphony
 

Was ist angesagt? (20)

Intro agile for PO's
Intro agile for PO'sIntro agile for PO's
Intro agile for PO's
 
Prioritise: Kanban Recipe for Success Step 5
Prioritise: Kanban Recipe for Success Step 5Prioritise: Kanban Recipe for Success Step 5
Prioritise: Kanban Recipe for Success Step 5
 
Building Better Products, June 2015
Building Better Products, June 2015Building Better Products, June 2015
Building Better Products, June 2015
 
Building a Culture of Product Experimentation by Optimizely PM
Building a Culture of Product Experimentation by Optimizely PMBuilding a Culture of Product Experimentation by Optimizely PM
Building a Culture of Product Experimentation by Optimizely PM
 
What Are the Product & Design Principles by FindMyPast PM
What Are the Product & Design Principles by FindMyPast PMWhat Are the Product & Design Principles by FindMyPast PM
What Are the Product & Design Principles by FindMyPast PM
 
What not to do when adopting Agile
What not to do when adopting AgileWhat not to do when adopting Agile
What not to do when adopting Agile
 
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
[CXL Live 16] Growth Hacking BS: Fixing Marketing One Truth at a Time by Morg...
 
ExumaTech Dev Process
ExumaTech Dev ProcessExumaTech Dev Process
ExumaTech Dev Process
 
Test Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew EakinTest Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew Eakin
 
The argument for agile development
The argument for agile developmentThe argument for agile development
The argument for agile development
 
Changing business of testing - Testing Assembly Helsinki 2014
Changing business of testing - Testing Assembly Helsinki 2014Changing business of testing - Testing Assembly Helsinki 2014
Changing business of testing - Testing Assembly Helsinki 2014
 
Why does agile need unit testing
Why does agile need unit testingWhy does agile need unit testing
Why does agile need unit testing
 
A/B Testing and the Infinite Monkey Theory
A/B Testing and the Infinite Monkey TheoryA/B Testing and the Infinite Monkey Theory
A/B Testing and the Infinite Monkey Theory
 
[HCMC STC Jan 2015] Creating a Better Testing Future
[HCMC STC Jan 2015] Creating a Better Testing Future[HCMC STC Jan 2015] Creating a Better Testing Future
[HCMC STC Jan 2015] Creating a Better Testing Future
 
What's a Product Manager?
What's a Product Manager? What's a Product Manager?
What's a Product Manager?
 
Why unvalidated assumption is the enemy of good product
Why unvalidated assumption is the enemy of good productWhy unvalidated assumption is the enemy of good product
Why unvalidated assumption is the enemy of good product
 
Optimizely Demo Deck
Optimizely Demo DeckOptimizely Demo Deck
Optimizely Demo Deck
 
Mock Interviewing with Salesforce's Product Manager
Mock Interviewing with Salesforce's Product ManagerMock Interviewing with Salesforce's Product Manager
Mock Interviewing with Salesforce's Product Manager
 
Quality is a Mindset
Quality is a MindsetQuality is a Mindset
Quality is a Mindset
 
Test Estimation Hacks: Tips, Tricks and Tools Webinar
Test Estimation Hacks: Tips, Tricks and Tools WebinarTest Estimation Hacks: Tips, Tricks and Tools Webinar
Test Estimation Hacks: Tips, Tricks and Tools Webinar
 

Ähnlich wie 10 Tactics for Building an Optimization Culture

Program Management 101: Best Practices from Optimizely-on-Optimizely
Program Management 101: Best Practices from Optimizely-on-OptimizelyProgram Management 101: Best Practices from Optimizely-on-Optimizely
Program Management 101: Best Practices from Optimizely-on-OptimizelyOptimizely
 
VWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation RoadmapVWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation RoadmapVWO
 
Agile Marketing Meetup: Moving Beyond the Marketing Plan So You Remain Relevant
Agile Marketing Meetup: Moving Beyond the Marketing Plan So You Remain RelevantAgile Marketing Meetup: Moving Beyond the Marketing Plan So You Remain Relevant
Agile Marketing Meetup: Moving Beyond the Marketing Plan So You Remain RelevantSean Ellis
 
Training Program Evaluation
Training Program EvaluationTraining Program Evaluation
Training Program EvaluationLaura Pasquini
 
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick SoChris Goward
 
Webinar slides with Paul Niven & Ben Lamorte “OKRs: Best Practices from the F...
Webinar slides with Paul Niven & Ben Lamorte “OKRs: Best Practices from the F...Webinar slides with Paul Niven & Ben Lamorte “OKRs: Best Practices from the F...
Webinar slides with Paul Niven & Ben Lamorte “OKRs: Best Practices from the F...Atiim, Inc.
 
Building Blocks of a strong Experimentation Program (1).pdf
Building Blocks of a strong Experimentation Program (1).pdfBuilding Blocks of a strong Experimentation Program (1).pdf
Building Blocks of a strong Experimentation Program (1).pdfVWO
 
Getting Started with Experimentation
Getting Started with ExperimentationGetting Started with Experimentation
Getting Started with ExperimentationOptimizely
 
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...pdx MindShare
 
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...pdxMindShare Admin
 
Startup Essentials: Marketing & Culture by Far Reach
Startup Essentials: Marketing & Culture by Far ReachStartup Essentials: Marketing & Culture by Far Reach
Startup Essentials: Marketing & Culture by Far ReachFar Reach
 
Testing Your Testing Program
Testing Your Testing ProgramTesting Your Testing Program
Testing Your Testing ProgramOptimizely
 
Testing Your Testing Program
Testing Your Testing ProgramTesting Your Testing Program
Testing Your Testing ProgramJill Martay
 
Scaling Engagement Agility
Scaling Engagement AgilityScaling Engagement Agility
Scaling Engagement AgilityDavid Dame
 
Hiring and Empowering Managers: How To Recruit & Train Managers To Maximize T...
Hiring and Empowering Managers: How To Recruit & Train Managers To Maximize T...Hiring and Empowering Managers: How To Recruit & Train Managers To Maximize T...
Hiring and Empowering Managers: How To Recruit & Train Managers To Maximize T...GreenhouseSoftware
 
Building Your Marketing Platform: 10 Do's and Don'ts of An Effective Marketin...
Building Your Marketing Platform: 10 Do's and Don'ts of An Effective Marketin...Building Your Marketing Platform: 10 Do's and Don'ts of An Effective Marketin...
Building Your Marketing Platform: 10 Do's and Don'ts of An Effective Marketin...Ellie Mirman
 
Onboarding Secrets: 10 Steps to Effective Onboarding and Boosting Your Retent...
Onboarding Secrets: 10 Steps to Effective Onboarding and Boosting Your Retent...Onboarding Secrets: 10 Steps to Effective Onboarding and Boosting Your Retent...
Onboarding Secrets: 10 Steps to Effective Onboarding and Boosting Your Retent...Tim Ruef
 

Ähnlich wie 10 Tactics for Building an Optimization Culture (20)

Program Management 101: Best Practices from Optimizely-on-Optimizely
Program Management 101: Best Practices from Optimizely-on-OptimizelyProgram Management 101: Best Practices from Optimizely-on-Optimizely
Program Management 101: Best Practices from Optimizely-on-Optimizely
 
VWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation RoadmapVWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation Roadmap
 
Agile Marketing Meetup: Moving Beyond the Marketing Plan So You Remain Relevant
Agile Marketing Meetup: Moving Beyond the Marketing Plan So You Remain RelevantAgile Marketing Meetup: Moving Beyond the Marketing Plan So You Remain Relevant
Agile Marketing Meetup: Moving Beyond the Marketing Plan So You Remain Relevant
 
Training Program Evaluation
Training Program EvaluationTraining Program Evaluation
Training Program Evaluation
 
Evidence-Based Management: This is how WE do it.
Evidence-Based Management: This is how WE do it.Evidence-Based Management: This is how WE do it.
Evidence-Based Management: This is how WE do it.
 
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
[Webinar] Scaling experimentation: 5 key pillars of maturity by Nick So
 
Webinar slides with Paul Niven & Ben Lamorte “OKRs: Best Practices from the F...
Webinar slides with Paul Niven & Ben Lamorte “OKRs: Best Practices from the F...Webinar slides with Paul Niven & Ben Lamorte “OKRs: Best Practices from the F...
Webinar slides with Paul Niven & Ben Lamorte “OKRs: Best Practices from the F...
 
Top Team Alignment
Top Team AlignmentTop Team Alignment
Top Team Alignment
 
Building Blocks of a strong Experimentation Program (1).pdf
Building Blocks of a strong Experimentation Program (1).pdfBuilding Blocks of a strong Experimentation Program (1).pdf
Building Blocks of a strong Experimentation Program (1).pdf
 
Getting Started with Experimentation
Getting Started with ExperimentationGetting Started with Experimentation
Getting Started with Experimentation
 
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
 
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
5 Strategies to Turn an Internship into a Career in Search Marketing | Kent L...
 
Startup Essentials: Marketing & Culture by Far Reach
Startup Essentials: Marketing & Culture by Far ReachStartup Essentials: Marketing & Culture by Far Reach
Startup Essentials: Marketing & Culture by Far Reach
 
Testing Your Testing Program
Testing Your Testing ProgramTesting Your Testing Program
Testing Your Testing Program
 
Testing Your Testing Program
Testing Your Testing ProgramTesting Your Testing Program
Testing Your Testing Program
 
Scaling Engagement Agility
Scaling Engagement AgilityScaling Engagement Agility
Scaling Engagement Agility
 
Hiring and Empowering Managers: How To Recruit & Train Managers To Maximize T...
Hiring and Empowering Managers: How To Recruit & Train Managers To Maximize T...Hiring and Empowering Managers: How To Recruit & Train Managers To Maximize T...
Hiring and Empowering Managers: How To Recruit & Train Managers To Maximize T...
 
Building Your Marketing Platform: 10 Do's and Don'ts of An Effective Marketin...
Building Your Marketing Platform: 10 Do's and Don'ts of An Effective Marketin...Building Your Marketing Platform: 10 Do's and Don'ts of An Effective Marketin...
Building Your Marketing Platform: 10 Do's and Don'ts of An Effective Marketin...
 
Lean Startup 301
Lean Startup 301Lean Startup 301
Lean Startup 301
 
Onboarding Secrets: 10 Steps to Effective Onboarding and Boosting Your Retent...
Onboarding Secrets: 10 Steps to Effective Onboarding and Boosting Your Retent...Onboarding Secrets: 10 Steps to Effective Onboarding and Boosting Your Retent...
Onboarding Secrets: 10 Steps to Effective Onboarding and Boosting Your Retent...
 

Mehr von Optimizely

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationOptimizely
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Optimizely
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing RightOptimizely
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleOptimizely
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Optimizely
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueOptimizely
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsOptimizely
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Optimizely
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Optimizely
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingOptimizely
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideOptimizely
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentOptimizely
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOptimizely
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...Optimizely
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyOptimizely
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueOptimizely
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' EyesOptimizely
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubOptimizely
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationOptimizely
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely
 

Mehr von Optimizely (20)

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical Teams
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team Offering
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server Side
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHub
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with Experimentation
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature Delivery
 

Kürzlich hochgeladen

Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfAnhNguyen97152
 
Plano de marketing- inglês em formato ppt
Plano de marketing- inglês  em formato pptPlano de marketing- inglês  em formato ppt
Plano de marketing- inglês em formato pptElizangelaSoaresdaCo
 
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...IMARC Group
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...Brian Solis
 
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003believeminhh
 
A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.mcshagufta46
 
MoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor PresentationMoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor Presentationbaron83
 
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfPDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfHajeJanKamps
 
To Create Your Own Wig Online To Create Your Own Wig Online
To Create Your Own Wig Online  To Create Your Own Wig OnlineTo Create Your Own Wig Online  To Create Your Own Wig Online
To Create Your Own Wig Online To Create Your Own Wig Onlinelng ths
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxWorkforce Group
 
Tata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakTata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakEditores1
 
Introduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxIntroduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxJemalSeid25
 
Fabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsFabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsWristbands Ireland
 
Lecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toLecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toumarfarooquejamali32
 
Anyhr.io | Presentation HR&Recruiting agency
Anyhr.io | Presentation HR&Recruiting agencyAnyhr.io | Presentation HR&Recruiting agency
Anyhr.io | Presentation HR&Recruiting agencyHanna Klim
 
7movierulz.uk
7movierulz.uk7movierulz.uk
7movierulz.ukaroemirsr
 
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...AustraliaChapterIIBA
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessAPCO
 

Kürzlich hochgeladen (20)

Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
 
Plano de marketing- inglês em formato ppt
Plano de marketing- inglês  em formato pptPlano de marketing- inglês  em formato ppt
Plano de marketing- inglês em formato ppt
 
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
 
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
 
A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.
 
MoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor PresentationMoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor Presentation
 
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfPDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
 
To Create Your Own Wig Online To Create Your Own Wig Online
To Create Your Own Wig Online  To Create Your Own Wig OnlineTo Create Your Own Wig Online  To Create Your Own Wig Online
To Create Your Own Wig Online To Create Your Own Wig Online
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
 
Tata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakTata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerak
 
Introduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxIntroduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptx
 
Fabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsFabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and Festivals
 
Lecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toLecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb to
 
Anyhr.io | Presentation HR&Recruiting agency
Anyhr.io | Presentation HR&Recruiting agencyAnyhr.io | Presentation HR&Recruiting agency
Anyhr.io | Presentation HR&Recruiting agency
 
7movierulz.uk
7movierulz.uk7movierulz.uk
7movierulz.uk
 
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
 

10 Tactics for Building an Optimization Culture

  • 1. 10 Tactics for Building an Optimization Culture OPTIMIZELY & BROOKS BELL 3.4.2014
  • 2. Welcome! Gregory Ng Chief Marketing Officer Brooks Bell @GregoryNg Kyle Rush Head of Optimization Optimizely @kylerush
  • 3. 10 Tactics for Developing a Successful Experimentation Culture
  • 4. What we’ll cover ... 1. Best practices from top testing organizations 2. The people behind testing 3. How to organize your program 4. How to keep the process manageable 5. How to include others and build excitement
  • 6. What does it mean to be data-driven? Progress is compelled by data, rather than by intuition or personal experience. Evidence-based decision making. Using data to inform key decisions.
  • 7. How data-driven are you, actually? • Do you dictate strategy without referencing data? • Do you rely on the person with the most experience to make the ultimate call? • Do you use words like “think” and “feel” when talking about strategy? • Does your team trust the data?
  • 8. How do you establish a successful testing culture?
  • 9. 10 Tactics 1. Clear organizational structure 2. Identify the right talent 3. Establish stakeholders 4. Solicit test ideas from everyone 5. Prioritize tests 6. Share results and successes 7. Build a knowledge base 8. Don’t rush into MVT tests 9. Balance learning and “wins” 10.Go beyond CRO
  • 10. #1 – Choose the right organization structure • No “one-size-fits-all” • One or multi-person team • Centralized or decentralized? Members of the Obama for America Team
  • 11. #1 – Choose the right organization structure • Army of One • Ringleader • Fellowship • Kingdom
  • 12. #2 – Pick the Right Talent • Qualitative and quantitative combination • Curiosity, creativity • People skills • Analytical mindset VP of Marketing E-Commerce Director Program Manager, Retail Optimizely’s customers come from a variety of backgrounds Product Manager
  • 13. #3 – Establish Stakeholders Early On • Identify the correct stakeholders • Secure an executive sponsor • Develop an approval process for tests • Communicate and share successes
  • 14. #4 – Solicit Test Ideas from Everyone • Make it easy to submit ideas • Capture interest after sharing results • Follow up for every idea • Create a scoreboard • Attribute experiments to team members Example: Optimizely Scorecard Vote on predicted winning variations
  • 15. #5 - Know how to Prioritize Tests ESTABLISH A SCORECARD •Relevance to business goals (KPIs) •Potential impact •Potential for learning •Level of effort •Time to significance and available traffic •Likelihood of implementation
  • 16. #6 – Share Results and Successes • • • • • Make it easy to understand Clarify the primary goal Share 100 percent of results, regardless of outcome Standing meetings Keep score Presenting A/B test findings at company all-hands
  • 17. #7 – Build a Knowledge Base • Gain broader insights • Lay a foundation for growth • Eliminate single point of failure • Increase accountability • Standardize processes and documentation
  • 18. #8 – Don’t Rush Into MVT Tests • Ideal for fine-tuning optimization • More complicated but less effective • Fewer learnings
  • 19. #9 – Balance Learning and “Wins” • • • • Always start with a hypothesis Never settle for a flat result Invitation for deep analysis Segment results, target accordingly
  • 20. #10 – Go Beyond CRO Optimization doesn’t need to occur in isolation. Almost anyone can test. For instance ...
  • 21. #10 – Go Beyond CRO • Designers Variation A Variation B Walk designers through A/B tests so they can inform their designs with data
  • 22. #10 – Go Beyond CRO • Engineers Launch of OFA 2012 New Donation API A/B testing the new platform revealed an architectural bug that was missed in QA
  • 23. #10 – Go Beyond CRO • Email and Content Marketers A/B Tested Emails and Landing Pages for Subscribers #dataculture
  • 24. 1. Clear organizational structure 2. Identify the right talent 3. Establish stakeholders 4. Solicit test ideas from everyone In summary ... 5. Prioritize tests 6. Share results and successes 7. Build a knowledge base 8. Don’t rush into MVT tests 9. Balance learning and “wins” 10.Go beyond CRO
  • 25. Thank you for reading!