Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Wil Stuckey
@quiiver
25 Million Members
25 Million Members
900,000 Shops
`
25 Million Members
900,000 Shops
1.4B Page Views / Month
Managing
Experimentation
Managing
Experimentation
Managing
Launches
1. Experiments
2. Rampups
3. Communications
Life Before Continuous
Deployments
Initial Idea &
Planning
Geocities-ize
Initial Idea &
Planning
Design &
Development
Initial Idea &
Planning
Deploy &
Launch!
Design &
Development
Pain?
Testing?
Communication?
How can we do better?
Living in a
Continuous
Deployment World
Total Deploys: 4,040
Total Commits: 45,447
Total Unique Authors: 220
2013 ... So far
Total Deploys: 4,040
Total Commits: 45,447
Total Unique Authors: 220
Average Deploys per day: 35.75
Average Authors per de...
That’s a lot of people
We are only halfway
through the year!
CHAOS
Config System
$server_config['my_awesome_feature'] = [
'enabled' => 10,
'admin' => 'on',
'groups' => [1234],
'users' => ['w...
Branching in Code
if (Feature::isEnabled('my_awesome_feature')) {
// display my_awesome_feature...
} else {
// continue on...
Typical Launch
Cycle
Initial Idea &
Planning
Initial Idea &
Planning
Design &
Development
Launch!
Initial Idea &
Planning
Design &
Development
Launch!
Internal Admin Launch
Initial Idea &
Planning
Design &
Development
Launch!
Internal Admin Launch
Public Prototype
Initial Idea &
Planning
Design &
Development
Launch!
Internal Admin Launch
Public Prototype
50% Experiment
• Small iterative changes
• Small iterative changes
• Product exists in prod as soon as possible
• Small iterative changes
• Product exists in prod as soon as possible
• Can be available for small groups for
testing and...
• Small iterative changes
• Product exists in prod as soon as possible
• Can be available for small groups for
testing and...
Because of the
constant change you
have to constantly
communicate.
Start Simple
Wiki + Email
• Works great in the
beginning
Wiki
• Works great in the
beginning
• Quickly becomes
overwhelming as you
increase your
concurrent launches.
Wiki
• Works great in the
beginning
• Quickly becomes
overwhelming as you
increase your
concurrent launches.
• Requires manual ...
FRUSTRATION
We can do better!
Frank Harris
@hirefrank
Launch Calendar
Launch Calendar
• Simple web app
• Collect structured meta-data about the
launch
• Central location for upcoming, current
...
Launch Calendar
Collect launch specific meta-data
Launch Calendar
• Owner
• Launch Date
• Internal Launch Date
• Team
• Expected Reaction
Collect launch specific meta-data
Launch Calendar
✉
Make your email work for you
✉
Launch Calendar
• Automated emails to product owners
around deadlines
• Daily emails to product groups containing
upcomi...
Launch Calendar
Make your email work for you
✉
Launch Calendar
• Highly utilized across the organization
• During the year+ that it was running we
recorded and tracked o...
“This beats the crap out
of the wiki page in every
conceivable way.” - real
feedback
What could we do better?
What would a more
integrated product look
like?
Can we automate more?
We can do better!
Catapult
A unified launch management tool.
“I believe that by changing
the ‘add to cart’ button to
blue we will increase
conversion rate.”
Catapult
$server_config['my_awesome_button'] = [
'enabled' => 50,
'admin' => 'on',
];
Remember this?
Hack, Deploy, Hack, Hack
Deploy!
Integration is
awesome.
Communication is
awesome.
Start Simple.
Build process that
enable you to Ship.
Never stop
deploying!
Thank You!
Wil Stuckey
@quiiver
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
Managing experimentation
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Design for Continuous Experimentation
Next
Upcoming SlideShare
Design for Continuous Experimentation
Next
Download to read offline and view in fullscreen.

Share

Managing experimentation

Download to read offline

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Managing experimentation

  1. 1. Wil Stuckey @quiiver
  2. 2. 25 Million Members
  3. 3. 25 Million Members 900,000 Shops `
  4. 4. 25 Million Members 900,000 Shops 1.4B Page Views / Month
  5. 5. Managing Experimentation
  6. 6. Managing Experimentation
  7. 7. Managing Launches
  8. 8. 1. Experiments 2. Rampups 3. Communications
  9. 9. Life Before Continuous Deployments
  10. 10. Initial Idea & Planning
  11. 11. Geocities-ize
  12. 12. Initial Idea & Planning Design & Development
  13. 13. Initial Idea & Planning Deploy & Launch! Design & Development
  14. 14. Pain?
  15. 15. Testing?
  16. 16. Communication?
  17. 17. How can we do better?
  18. 18. Living in a Continuous Deployment World
  19. 19. Total Deploys: 4,040 Total Commits: 45,447 Total Unique Authors: 220 2013 ... So far
  20. 20. Total Deploys: 4,040 Total Commits: 45,447 Total Unique Authors: 220 Average Deploys per day: 35.75 Average Authors per deploy: 7.01 Average Commits per deploy: 11.25 2013 ... So far
  21. 21. That’s a lot of people
  22. 22. We are only halfway through the year!
  23. 23. CHAOS
  24. 24. Config System $server_config['my_awesome_feature'] = [ 'enabled' => 10, 'admin' => 'on', 'groups' => [1234], 'users' => ['wstuckey', 'dmckinley'], ];
  25. 25. Branching in Code if (Feature::isEnabled('my_awesome_feature')) { // display my_awesome_feature... } else { // continue on }
  26. 26. Typical Launch Cycle
  27. 27. Initial Idea & Planning
  28. 28. Initial Idea & Planning Design & Development Launch!
  29. 29. Initial Idea & Planning Design & Development Launch! Internal Admin Launch
  30. 30. Initial Idea & Planning Design & Development Launch! Internal Admin Launch Public Prototype
  31. 31. Initial Idea & Planning Design & Development Launch! Internal Admin Launch Public Prototype 50% Experiment
  32. 32. • Small iterative changes
  33. 33. • Small iterative changes • Product exists in prod as soon as possible
  34. 34. • Small iterative changes • Product exists in prod as soon as possible • Can be available for small groups for testing and feedback.
  35. 35. • Small iterative changes • Product exists in prod as soon as possible • Can be available for small groups for testing and feedback. • Easy to communicate milestones because they happen as part of the development process.
  36. 36. Because of the constant change you have to constantly communicate.
  37. 37. Start Simple
  38. 38. Wiki + Email
  39. 39. • Works great in the beginning Wiki
  40. 40. • Works great in the beginning • Quickly becomes overwhelming as you increase your concurrent launches. Wiki
  41. 41. • Works great in the beginning • Quickly becomes overwhelming as you increase your concurrent launches. • Requires manual labor to keep stakeholders informed Wiki
  42. 42. FRUSTRATION
  43. 43. We can do better!
  44. 44. Frank Harris @hirefrank
  45. 45. Launch Calendar
  46. 46. Launch Calendar • Simple web app • Collect structured meta-data about the launch • Central location for upcoming, current and past launch information
  47. 47. Launch Calendar Collect launch specific meta-data
  48. 48. Launch Calendar • Owner • Launch Date • Internal Launch Date • Team • Expected Reaction Collect launch specific meta-data
  49. 49. Launch Calendar ✉ Make your email work for you
  50. 50. ✉ Launch Calendar • Automated emails to product owners around deadlines • Daily emails to product groups containing upcoming and recently launched launches Make your email work for you
  51. 51. Launch Calendar Make your email work for you ✉
  52. 52. Launch Calendar • Highly utilized across the organization • During the year+ that it was running we recorded and tracked over 275 launches How did it go?
  53. 53. “This beats the crap out of the wiki page in every conceivable way.” - real feedback
  54. 54. What could we do better?
  55. 55. What would a more integrated product look like?
  56. 56. Can we automate more?
  57. 57. We can do better!
  58. 58. Catapult A unified launch management tool.
  59. 59. “I believe that by changing the ‘add to cart’ button to blue we will increase conversion rate.”
  60. 60. Catapult
  61. 61. $server_config['my_awesome_button'] = [ 'enabled' => 50, 'admin' => 'on', ]; Remember this?
  62. 62. Hack, Deploy, Hack, Hack Deploy!
  63. 63. Integration is awesome.
  64. 64. Communication is awesome.
  65. 65. Start Simple.
  66. 66. Build process that enable you to Ship.
  67. 67. Never stop deploying!
  68. 68. Thank You!
  69. 69. Wil Stuckey @quiiver
  • PaulRichard32

    Feb. 14, 2021
  • muhammadajib56

    Mar. 16, 2020
  • SergeySimonyan

    Jan. 9, 2015
  • spovlot

    Dec. 15, 2014
  • barrettcook

    Jun. 24, 2014
  • AmitBagade

    Apr. 14, 2014
  • kristw

    Apr. 11, 2014
  • llch

    Mar. 4, 2014
  • cullmann

    Jul. 26, 2013
  • tahoemgo

    Jun. 28, 2013
  • petervanhaken1

    Jun. 28, 2013
  • bertrandtornil

    Jun. 28, 2013
  • geloman

    Jun. 28, 2013
  • fmlalawyers

    Jun. 28, 2013
  • octablogs

    Jun. 27, 2013
  • adamnash

    Jun. 27, 2013
  • AndyDavies

    Jun. 27, 2013

Views

Total views

16,529

On Slideshare

0

From embeds

0

Number of embeds

913

Actions

Downloads

65

Shares

0

Comments

0

Likes

17

×