Devops Management is a topic discussed in the halls of conferences and few managers. This talk will focus on the topic of management in a highly collaborative and cooperative environment, specifically one that is rapidly growing with a focus on continuous development/deployment

1. DevOps Management:
Herding Cats at
Michael Rembetsy
Director, Operations Engineering
@mrembetsy

2. What is Etsy?
“Etsy is the marketplace we make together. We enable
people anywhere to easily build and directly exchange with
independent, creative businesses”

4. The Numbers!

5. The Numbers!
• 15 million members

6. The Numbers!
• 15 million members
• 875,000 active sellers

7. The Numbers!
• 15 million members
• 875,000 active sellers
• Sellers in over 150 countries

8. The Numbers!
• 15 million members
• 875,000 active sellers
• Sellers in over 150 countries
• 2011 sellers grossed 525 million in sales

9. The Numbers!
• 15 million members
• 875,000 active sellers
• Sellers in over 150 countries
• 2011 sellers grossed 525 million in sales
• 1.4 billion page views a month and growing

10. Who am I?

11. Who am I?
• Responsible for day to day collaboration
and operations (datacenter / corporate)

12. Who am I?
• Responsible for day to day collaboration
and operations (datacenter / corporate)
• I am a mentor

13. Who am I?
• Responsible for day to day collaboration
and operations (datacenter / corporate)
• I am a mentor
• I enable

14. Who am I?
• Responsible for day to day collaboration
and operations (datacenter / corporate)
• I am a mentor
• I enable
• Remove barriers and encourage shipping

16. Culture

18. The old Culture didn’t
work at Etsy

20. So what is the new way?

21. Starts with Hiring

22. Starts with Hiring
• Manager != smartest person on the team,
hire smart people

23. Starts with Hiring
• Manager != smartest person on the team,
hire smart people
• You need to be more than just a
technologist

24. Starts with Hiring
• Manager != smartest person on the team,
hire smart people
• You need to be more than just a
technologist
• Experiences count for managers and team
members

27. Required

29. Principles for a DevOps
Managers

30. Principles for a DevOps
Managers
• Accept failures not lower standards

31. Principles for a DevOps
Managers
• Accept failures not lower standards
• Доверяй, но проверяй

32. Principles for a DevOps
Managers
• Accept failures not lower standards
• Доверяй, но проверяй
• Celebrate success and failures (blameless
post mortems)

33. Principles for a DevOps
Managers
• Accept failures not lower standards
• Доверяй, но проверяй
• Celebrate success and failures (blameless
post mortems)
• Be open minded

34. Principles for a DevOps
Managers
• Accept failures not lower standards
• Доверяй, но проверяй
• Celebrate success and failures (blameless
post mortems)
• Be open minded
• Move fast (collaboration and deployment)

35. The Team

36. The Team
• Flexible. Able to work in a dynamic
environment anywhere

37. The Team
• Flexible. Able to work in a dynamic
environment anywhere
• There is no single variable to success.
There are many.

38. The Team
• Flexible. Able to work in a dynamic
environment anywhere
• There is no single variable to success.
There are many.
• Team members and managers must know
they are liaisons for all other teams

39. How do I manage my
team during...

40. Non-Stressful Times

41. Stressful Times

42. The Same Way!

43. The Same Way!
• Failures / outages

44. The Same Way!
• Failures / outages
• Personality conflicts

45. The Same Way!
• Failures / outages
• Personality conflicts
• Personal issues

46. The Same Way!
• Failures / outages
• Personality conflicts
• Personal issues
• Last minute whoopsies

47. The Same Way!
• Failures / outages
• Personality conflicts
• Personal issues
• Last minute whoopsies
• Unexpected context switching (work)

48. One on One

49. “What’s the point? We talk all the time in IRC?”
“I don’t have time, just email me an update.”

50. NO!!!!!

51. NO!!!!!
Provide a scheduled outlet for talk (30
minutes minimum)

52. NO!!!!!
Provide a scheduled outlet for talk (30
minutes minimum)
Respect their time and yours, show up on
time and prepared to discuss the last week

53. http://www.randsinrepose.com/archives/2010/09/22/
the_update_the_vent_and_the_disaster.html

54. Humor

58. Freedom

59. Engineers are creative.
So let them be creative....

60. Freedom but with
Guidelines
• Communication: send everyone updates
• Ask for help if you need it
• Ask for direction on work
• Day to day work is done (queue, job
description)

61. Open Door Policy

62. Problems Managing People

63. Problems Managing People

64. Problems Managing People
• You will have them and each is different

65. Problems Managing People
• You will have them and each is different
• Stay calm

66. Problems Managing People
• You will have them and each is different
• Stay calm
• Talk to your peers for advice

67. Problems Managing People
• You will have them and each is different
• Stay calm
• Talk to your peers for advice
• Think outside the box

68. Problems Managing People
• You will have them and each is different
• Stay calm
• Talk to your peers for advice
• Think outside the box
• Don’t fear having hard conversations.
These make you a better manager

69. “the job of a manager is to be willing to have a
hard conversation, any time during the week
with anyone anywhere.” Marc Hedlund

70. We also changed our
workflow in operations

71. Why?
We weren’t shipping enough
Context switching = low morale
“what are you working on?”

72. Kanban

73. Kanban
• Provided the whole organization
Transparency into what we were doing

74. Kanban
• Provided the whole organization
Transparency into what we were doing
• Less context switching = better morale

75. Kanban
• Provided the whole organization
Transparency into what we were doing
• Less context switching = better morale
• Better throughput of work

76. Kanban
• Provided the whole organization
Transparency into what we were doing
• Less context switching = better morale
• Better throughput of work
• Better Prioritization

77. How Does it flow?

82. We use a virtual board

83. Give them tools
&
Encourage building
collaborative tools!

84. Devops Tools - managers
• Jira- tickets All can see
• Kanban - workflow that you just saw
• but wait theres more.....

85. Metrics

86. Metrics
• We have ~280K now and growing!

87. Metrics
• We have ~280K now and growing!
• Use open source tools (ganglia, graphite)

88. Metrics
• We have ~280K now and growing!
• Use open source tools (ganglia, graphite)
• Dashboards are key

91. Monitoring
• We use tools like Nagios, Pagerduty and
Pingdom.

92. Monitoring
• We use tools like Nagios, Pagerduty and
Pingdom.
• More than ops are on call

93. Monitoring
• We use tools like Nagios, Pagerduty and
Pingdom.
• More than ops are on call
• ops on call rotation

94. Monitoring
• We use tools like Nagios, Pagerduty and
Pingdom.
• More than ops are on call
• ops on call rotation
• dev on call rotation

95. Monitoring
• We use tools like Nagios, Pagerduty and
Pingdom.
• More than ops are on call
• ops on call rotation
• dev on call rotation
• search on call rotation

96. Human escalation

97. Asset Management

99. Chef deployment is
transparent to everyone
Deployed Chef Change

100. That wasn’t enough
collaboration

101. Eventinator

103. Let’s review

105. • Culture
• One on One meeting
• Humor
• Freedom to be creative and explore
• Open Door Policy
• Encourge the use and building of
collabrative tools
• DevOps Princples for managers
• Be transparent with your work to everyone

106. My Influences
• Michael Lopp (http://www.randsinrepose.com)
• Kate Matsudaira (http://www.katemats.com)
• Chad Dickerson (http://blog.chaddickerson.com)
• John Allspaw (http://kitchensoap.com/)
• Marc Hedlund (http://blog.precipice.org/)
• Kellan Elliot-McCrea (https://twitter.com/#!/kellan)

107. Thank you for listening
Questions?