How We Think About ITPreordains Our Outcomes: DevOps And More…

1. How We Think About IT
Preordains Our Outcomes:
DevOps And More…
Gene Kim
Session ID:

2. Act I: IT Ops Fixing Fragile Artifacts

4. Act 2: The Product Managers

5. Act 3: The Developers

8. Act 4: Dev And IT Ops At War
8

10. The Downward Spiral
Operations Sees… Dev Sees…
 Fragile applications are prone to  More urgent, date-driven projects
failure put into the queue
 Long time required to figure out “which  Even more fragile code (less
bit got flipped” secure) put into production
 Detective control is a salesperson  More releases have increasingly
“turbulent installs”
 Too much time required to restore
service  Release cycles lengthen to
amortize “cost of deployments”
 Too much firefighting and unplanned
work  Failing bigger deployments more
difficult to diagnose
 Urgent security rework and
remediation  Most senior and constrained IT
ops resources have less time to
 Planned project work cannot complete fix underlying process problems
 Frustrated customers leave  Ever increasing backlog of work
 Market share goes down that cold help the business win
 Business misses Wall Street  Ever increasing amount of
commitments tension between IT
Ops, Development, Design…
 Business makes even larger promises
to Wall Street

11. The IT Core Chronic Conflict
 Every IT organization is pressured to
simultaneously:
 Respond more quickly to urgent business needs
 Provide stable, secure and predictable IT service
Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and
author of The Goal, has written extensively on the theory and practice of identifying and resolving
11 core, chronic conflicts.

12. Every Company Is An IT Company…
 95% of all capital projects have an IT
component…
 50% of all capital spending is technology-related
Where we need to
be…
IT is always in the
way
(again…)
We are here…

13. There Must Be A Better
Way…
13

14. Source: John Allspaw

15. Source: John Allspaw

16. Source: John Allspaw

17. Source: John Allspaw

18. Source: John Jenkins, Amazon.com

19. The First Way:
Systems Thinking

20. The First Way:
Systems Thinking (Left To Right)
 Understand the flow of work
 Always seek to increase flow
 Never unconsciously pass defects downstream
 Never allow local optimization to cause global
degradation
 Achieve profound understanding of the system

21. “Annual business planning sessions can be
madding. They think IT Operations is an „all you
can eat buffet.‟”
-Ben Rockwood,
Director Systems Engineering,
Joyent

22. Practice #1: Define The Work and Make It
Visible
 Business projects (e.g., new order system)
 Internal IT projects (e.g., Puppet automation)
 Changes (e.g., deploys, improve database
performance)
 Unplanned work (e.g., site down, site impaired)
22

23. Day 2: PMO Meeting

24. Practice #2: Create One Step Environment
Creation Process
 Make environments available early in the
Development process
 Make sure Dev builds the code and environment
at the same time
 Create a common Dev, QA and Production
environment creation process

25. Change the Agile sprint policy:
“At the end of each sprint, we must have working
code and the environment it runs in!!

26. The First Way:
Outcomes
 Creating single repository for code and environments
 Determinism in the release process
 Consistent Dev, QA, Int, and Staging environments, all
properly built before deployment begins
 Decreased cycle time
 Reduce deployment times from 6 hours to 45 minutes
 Refactor deployment process that had 1300+ steps
spanning 4 weeks
 Faster release cadence

27. The Second Way:
Amplify Feedback Loops

28. The Second Way:
Amplify Feedback Loops (Right to Left)
 Understand and respond to the needs of all
customers, internal and external
 Shorten and amplify all feedback loops: stop the
line when necessary
 Create quality at the source
 Create and embed knowledge where we need it

29. “We found that when we woke up developers at
2am, defects got fixed faster than ever”
-Patrick Lightbody,
CEO, BrowserMob

30. Pattern #3: Embed Dev Into IT Ops
 Embed Dev into IT Ops incident escalation
process
 Invite Dev to post-mortems/root cause analysis
meeting
 Have Dev and Infosec cross-train IT Operations
 Ensure application monitoring/metrics to aid in
Ops and Infosec work (e.g., incident/problem
management)

31. The Second Way:
Outcomes
 Defects and security issues getting fixed faster
than ever
 Reusable Ops and Infosec user stories now part
of the Agile process
 All groups communicating and coordinating
better
 Everybody is getting more work done

32. The Third Way:
Culture Of Continual Experimentation And
Learning

33. The Third Way:
Culture Of Continual Experimentation And
Learning
 Foster a culture that rewards:
 Experimentation (taking risks) and learning from
failure
 Repetition is the prerequisite to mastery
 Why?
 You need a culture that keeps pushing into the danger
zone
 And have the habits that enable you to survive in the
danger zone

34. Break Things Early And Often
 “Do painful things more frequently, so you can
make it less painful… We don‟t get pushback
from Dev, because they know it makes rollouts
smoother.”
-- Adrian Cockcroft, Architect, Netflix

35. Pattern #5: Inject Failures Often

36. You Don’t Choose Chaos Monkey…
Chaos Monkey Chooses You

37. Pattern #6: Break Things Before Production
 Enforce consistency in code, environments and
configurations across the environments
 Add your ASSERTs to find
misconfigurations, enforce https, etc.
 Add static code analysis to automated
continuous integration and testing process

38. Pattern #6: Allocate 20% Of Cycles To
Technical Debt Reduction

39. Recognize Compounding Technical Debt…

40. That Gets Worse…

41. And Fixing It…
Source: Pingdom

42. An Innovation Culture
“By installing a rampant innovation culture, they
now do 165 experiments in the three months of tax
season.
Our business result? Conversion rate of the
website is up 50 percent. Employee result?
Everyone loves it, because now their ideas can
make it to market.”
--Scott Cook, Intuit Founder
42

44. Why Do I Think This Is
Important?
44

45. The Downward Spiral
Operations Sees… Dev Sees…
 Fragile applications are prone to  More urgent, date-driven projects
failure put into the queue
 Long time required to figure out “which  Even more fragile code (less
bit got flipped” secure) put into production
 Detective control is a salesperson  More releases have increasingly
“turbulent installs”
 Too much time required to restore
service  Release cycles lengthen to
amortize “cost of deployments”
 Too much firefighting and unplanned
work  Failing bigger deployments more
difficult to diagnose
 Urgent security rework and
remediation  Most senior and constrained IT
ops resources have less time to
 Planned project work cannot complete fix underlying process problems
 Frustrated customers leave  Ever increasing backlog of work
 Market share goes down that cold help the business win
 Business misses Wall Street  Ever increasing amount of
commitments tension between IT
Ops, Development, Design…
 Business makes even larger promises
to Wall Street

46. The Three Ways: Some Patterns
First Way Second Way Third Way
Define The Wake Up Break Things Early
Work And Make Developers And Often
It Visible
Make Embed Dev Into IT Reserve 20% Of
Environments Operations Cycles For
Available Early Technical Debt
Reduction
46

47. Help The Business Win…

48. With Support From Your Peers…

49. And Do More With Less Effort…

50. 50

52. When IT Fails: A Business Novel and
The DevOps Cookbook
 Coming January 15, 2013 and Q1 2013
 “The lessons in When IT Fails might just save your business if IT fails
for you. Every IT executive should share this book with their business
peers.” -James Turnbull, VP Operations, Puppet Labs and author
of “Pro Puppet”
“The greatest IT management book of our generation.” –Branden
Williams, CTO Marketing, RSA
“This book will have a profound effect on IT, just as The Goal did for
manufacturing.‟ - Jez Humble, co-author of the Jolt award-winning
book Continuous Delivery, and Principal at ThoughtWorks
Studios.

53. Our Mission: Positively Impact The Lives Of
One Million IT Workers By 2017
 For these slides, the “Top 10 Things You
Need To Know About DevOps,” Rugged
DevOps resources, and updates on the
book:
Sign up at http://itrevolution.com
Email genek@realgenekim.me
 Or text “[email] 74730” to
+1 (858) 598-3980
 Visit:
http://www.instantcustomer.com/go/7473
0