There was a time not long ago when Etsy was laden with barriers, silos, broken communication, and noncooperation. This talk will focus on the various stages of Etsy's cultural development from the early days to present. We will tell of how Etsy overcame numerous challenges and built a strong company culture while continuing to scale.

1. Continuously Deploying
Culture
Scaling Culture at
Michael Rembetsy, Director of Operations Engineering (@mrembetsy)
Patrick McDonnell, Senior Operations Engineer (@mcdonnps)
October 4, 2012
Velocity Europe

2. About Us
• Operations team at Etsy totals twelve
• Engineering totals 125
• Company totals about 350
• 15 million members in over 150 countries
• Over 800,000 sellers
• 1.4 billion page views monthly

3. Disclaimer
A + B != Culture

4. The Beginning

5. Silos and Barriers
• Etsy employs 30 to 35
• About half are engineers
• Siloed culture, barriers to collaboration in engineering
• “Sprouter”
• Designed to prevent engineers from directly touching databases

6. Management Shake-up
• Maria Thomas promoted to CEO
• Understands that community is of utmost importance
• Begins to prioritize culture that supports community
• Chad Dickerson brought on as CTO
• Starts to bring focus to engineering team

7. Bring the Pain

9. Opening Communications
• fix.etsy.com
• First time exposing engineering to community
• Scheduled maintenance posted ahead of time
• Outage follow-ups
• Culturally, technical debt needs to be paid off

11. Opening Communications
• fix.etsy.com
• First time exposing engineering to community
• Scheduled maintenance posted ahead of time
• Outages followed upon
• Culturally, technical debt needs to be paid off

12. 2008 Takeaways
• WTF did I get myself into?
• Low or no impact projects
• Downtime is accepted / needed
• Community is most important
• Lucky to make it through holiday season

13. 2008 End of Year Snapshot
• Gross Merchandise Sales: $87.3 million
• Visits: 163 million
• Unique Visits: 47.7 million

14. 2008 Action Items
• Gain support from the top and bottom to change culture
• Increase transparency both within the organization and to the public
• Pay back technical debt as soon as possible

15. Sea Change
• Major shift in culture
• People leave or are let go
• Too many remotes
• Teams centralized in Brooklyn
• Need “DevOps” culture before remotes are manageable

16. Building the Foundation
• DevTools team created
• Develops first iteration of Deployinator
• Stabilizes the deploy chain through dev, QA, and prod environments
• Infrastructural overhaul
• Move from lighttpd to Apache starts
• Added monitoring and graphing
• Better metrics exposure to non-technical divisions

19. Building the Foundation
• DevTools team created
• Develops first iteration of Deployinator
• Stabilizes the deploy chain through dev, QA, and prod environments
• Infrastructural overhaul
• Move from lighttpd to Apache starts
• Added monitoring and graphing
• Better metrics exposure to non-technical divisions

20. Hiring Push
• Only two operations engineers and too few local developers
• Concentration on sourcing talent to keep pace with growth
• Move from Downtown Brooklyn to larger space in DUMBO
• More accessible to non-Brooklynites
• More mature tech and art community

21. Internal Improvement
• Stand-up meetings
• Time-consuming, but necessary to improve communication
• Inter-team collaboration
• Ops involved in decision-making for dev projects and vice versa
• Internal stability much stronger, damage control period ends
• Network solidified, infrastructural foundation allows for future growth

22. Stability Arrives
• Everyone wants to collaborate from top down and bottom up
• Not just upper management pushing down
• Everyone shapes culture, suggestions always welcome
• People happy to come to work, contribute beyond their job description
• No more scheduled downtime, site remains up as much as possible
• Master database purchased as a stopgap for holiday capacity

23. 2009 Takeaways
• Foundation solidified
• Technology
• Human capital
• Beginning of “DevOps” culture
• Berlin Wall falls

24. 2009 End of Year Snapshot
• Gross Merchandise Sales: $177 million (102.6%)
• Visits: 320 million (96.8%)
• Unique Visits: 92.9 million (94.8%)
• Page Views: 6.45 billion

25. 2009 Action Items
• Stabilize the parts of your organization which create thrashing
• Hire staff that will make a difference
• Think about collaborative processes
• When dealing with staffing constraints, prioritize projects by impact
• Get things done (later “Just Ship”)

26. Renewed Energy

28. .etsy.com

29. 40
30
20
10
Very end of 2009
Today

30. Standardization Effort
• PHP
• Use it and nothing else
• Everyone should be able to read and rewrite your code
• MySQL becomes database of choice
• Backfill of PostgreSQL tables to MySQL shards begins

31. If it Moves, Graph it
• Graphing tools
• Ganglia, Graphite, FITB, Weathermap
• Monitoring
• Nagios, Naglite
• Increased focus on work/life balance
• We removed alerts! WHAT?

34. If it Moves, Graph it
• Graphing tools
• Ganglia, Graphite, FITB, Weathermap
• Monitoring
• Nagios, Naglite
• Increased focus on work/life balance
• We removed alerts! WHAT?

36. Management Ideals
• Accept failures but not lower standards
• Доверяй, но проверяй (trust, but verify)
• Blameless post-mortems
• Welcome one-on-ones (http://bit.ly/cCWMqr)
• Career planning
• Happy company = happy community

37. Engineering Processes
• Developer on call
• A/B Testing
• Prototypes
• Feature flags and ramp-ups
• Schema Change Thursday

38. 2010 Takeaways
• Paring down the number of technologies used for development
• Focus on technical visibility throughout the organization
• Developers take responsibility for code release
• Freedom to hire as needed
• New focus on work life balance

39. 2010 End of Year Snapshot
• Gross Merchandise Sales: $307 million (73.4%)
• Visits: 534 million (66.7%)
• Unique Visits: 147 million (58.2%)
• Page Views: 9.25 billion (43.5%)

40. 2010 Action Items
• Don’t guess at what’s wrong; graph it, monitor it, and find out
• Empower developers with responsibilities
• Have clear, documented standards and practices
• Keep human management a priority

41. The Reaping
• Death of non-standard technologies
• Mongo, Scala, CoffeeScript, etc. removed from production systems
• Sprouter is eradicated
• No more Python
• Removal of a major barrier to full developer accountability
• Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)

43. The Reaping
• Death of non-standard technologies
• Mongo, Scala, CoffeeScript, etc. removed from production systems
• Sprouter is eradicated
• No more Python
• Removal of a major barrier to full developer accountability
• Check out Ross Snyder’s talk at Surge 2011 (http://bit.ly/po8zIj)

44. Organizational Change
• Management becomes more engineering focused
• Chad becomes CEO
• Kellan is promoted to CTO
• John is promoted to SVP of Technical Operations

45. Technical Contribution
• Big push to contribute to open source and technical community
• Deployinator, Statsd, Logster, many more
• Three annual goals for every engineer
• Speak at a conference
• Write a blog post
• Open source code

46. Still Scaling

47. Configuration Management

49. And Still Scaling Culture
• Engineering culture decides git is better for our working style than svn
• Product development efforts are refocused on high-impact products such as search
ads
• Ops works to improve signal-to-noise ratio of alerts and develops internal tools such
as Schemanator
• Focus on security
• SCRAM team, Security and Fraud team
• Check out Nick Galbreath’s talk at DevOpsDays Austin (http://slidesha.re/IMaavq)

50. Happy Holidays
• Game days to test failures before they happen
• Capacity planning is easier
• More dashboards (framework now on GitHub!)
• Improved weekly financial reporting

52. Happy Holidays
• Game days to test failures before they happen
• Capacity planning is easier
• More dashboards (framework now on GitHub!)
• Improved weekly financial reporting

53. 2011 Takeaways
• Year of the tool
• Statsd, Deployinator, Supergrep v2, Logster, Schemanator, FITB
• DevTools adds three engineers
• Automation reaches maturity, few surprises
• Focus on security allows PCI infrastructure to be completed in 6
weeks
• Engineering matures, both platform and people

54. 2011 End of Year Snapshot
• Gross Merchandise Sales: $526 million (71.4%)
• Visits: 895 million (67.7%)
• Unique Visits: 237 million (61.1%)
• Page Views: 12.9 billion (39.3%)

55. 2011 Action Items
• Senior management at a tech company should be technology-
focused
• Implement configuration management and automation as soon as
possible (saves headaches later)
• As technical staff increases, continue to focus on projects that
matter
• New technical challenges do not dictate cultural shift

56. Massive Growth
• Explosive growth in hiring
• Major organizational changes around product
• Increased focus on community
• High-impact products (shipping labels, gift cards)
• Ramp-up of internationalization projects
• Unique engineering challenges force non-standard technology
• Redis
• Virtualized CI test slaves

57. Spreading Information
• Invitation to other teams to stop by the office for a chat
• Ops becomes more involved in external informational exchange
• Code as Craft blog posts
• Rapid scaling of personnel, no one knows everyone
• Remotes come in to Brooklyn more often

60. Work in Progress
• Developer boredom curbed by allowing transfers between teams
• Even between divisions (such as engineering -> product)
• Developers now read data from prod databases for development
• Removes one of the last anomalies between development and
production
• Front End Performance team forms to minimize site load times

61. 2012 Takeaways
• Organizational resilience allows for total focus on community
• Improvement on international product
• Increased transparency to the public
• Focus on informational exchange, both internally and externally
• Open source all the things

62. 2012 Action Items
• Know when not to try something
• Focus on performance early
• Allow dynamic allocation of resources
• Don’t allow size to dictate culture

63. Future Predictions
• “Wait till you hit 500 people, that’s when everything falls apart”
• Etsy will not fall apart, but will change
• Management supports and responds to cultural shifts
• Freedom of movement between job functionalities
• Trust that change is for the better

64. Future Predictions
• Will create better and innovative ways of communicating as we scale
• Building tools is the Etsy way
• Continue to influence corporate culture
• B Corp
• Employee happiness

65. Office Hours
Sponsor Pavilion Table A
2:00 to Last Call at the Bar
Michael Rembetsy (@mrembetsy)
Patrick McDonnell (@mcdonnps)
https://github.com/etsy