As we architect our systems for greater demands, scale, uptime, and performance, the hardest thing to control becomes the environment in which we deploy and the subtle but crucial interactions between complicated systems. Chaos Patterns help us establish and implement a virtuous cycle that let’s us both prove & improve our system along each of these dimensions before the inevitable happens. While it may seem reckless or counter-intuitive, our experience has proven that it’s a matter of how and when (not if) we will learn about the limitations and failure modes of the system. This is the story of the pitfalls we encountered, and how, through architecture, convention and common sense, we managed to build an infrastructure that is "Always Up" from the end user perspective and incredibly economical to build, scale & operate; using chaos testing, we learn more about how our system fails from a 10 second controlled failure than a multi-hour uncontrolled outage. In this session we will cover various implementation techniques, available to any developer & operator, which will vastly increase the resilience of your systems and provide a superior end user experience; from optimizing your use of DNS for failure, to configuring your CDN to have your back, to synthetic responses and expected database outages. But why stop there? Netflix has pioneered a culture and suite of tools that actively injects ‘once in a blue moon’ failures into its production systems, which lets you battle test your resilience design and let developers & operators sleep comfortably at night knowing their systems are able to handle even the worst of worst case scenarios.

1. CHAOS PATTERNS
Architecting for failure in distributed systems
Bruce Wong - @bruce_m_wong / Jos Boumans - @jiboumans
http://www.soponderando.com.br/

2. http://fotos.subefotos.com/7a6b3e6df9453d5adf150087e5300834o.jpg
How to measure
everything
Architecting
in AWS for
resilience & cost
www.slideshare.net/jiboumans/aws-architecting-for-resilience-cost-at-scale
http://www.slideshare.net/jiboumans/how-to-measure-everything-a-million-metrics-per-second-with-minimal-developer-overhead

3. VP of Operations &
Infrastructure
http://www.krux.com/
3 Billion Users

4. ABOUT BRUCE
2010 2015
Software Engineer
Insight Engineering
Senior Engineering Manager
Chaos Engineering
Prosumers Consumers Enterprise
http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html

5. A LOT OFTRAFFIC
http://www.americapictures.net/buenos-aires-traffic-city-night-argentina.html

6. http://grandprix247.com/2012/09/03/spa-pile-up-renews-focus-on-formula-1-safety-matters/
REAL WORLD FAILURES

7. SEPTEMBER 20TH, 2015
Also:April 21, 2011 - June 29, 2012 - October 22, 2012 -
December 24, 2012 - August 26, 2013 <out of space>
https://twitter.com/iamDeveloper/status/645659734767329281 https://aws.amazon.com/message/5467D2/

8. ISOLATION & CONTAINMENT
Ideally limit failure to a single service
Stop it from spreading
http://businessnerds.wordpress.com/2011/05/28/so-far-so-good…-the-review/

9. So#ware,)8)
Automa/on,)4)
Process,)14)
Amazon"Cloud"Major"Outage"7"Issues"Categories"
https://steamcommunity.com/app/620/
http://fotos.subefotos.com/7a6b3e6df9453d5adf150087e5300834o.jpg
AWS Root Cause
Analysis over time
http://www.slideshare.net/rahultyagi50999/amazon-cloud-major-outages-analysis

10. Humans,
Software,
Processes
All likely causes
of failure
Isolation
Unlikely
2 - 4x
Yearly frequency of
catastrophic failure

11. THERE ARE DOWNSIDES
http://modernsavage.hubpages.com/hub/10-springfield-shopper-headlines

12. Complex Systems
Difficult to model, not feasible to simulate at scale

13. Products evolve
Embrace change

14. Resilience is a feature
Embrace failure

15. TWITTER

16. CORE EXPERIENCE
Enriched with optional enhancements

17. http://usa.streetsblog.org/category/issues-campaigns/air-quality/
NAVIGATING THE CHAOS

18. FALLBACK PATTERNS
“Expect the Unexpected”
http://blabitcanada.com/category/twitter-2/

19. BASIC API CALL
3 potential points of failure

20. FALLBACK PATTERNS
The cost of resilience should be accuracy or latency
http://redis.io/
http://memcached.org/
http://varnish-cache.org/

21. https://www.flickr.com/photos/ichijo2009/8501266124
ENSURING DATA ACCESS

22. CAP THEOREM APPLIES
Your choice: sacrifice availability or consistency.
Orange is a lie.
RDBMS
BigTable Based
Master / Slave based
CouchDB
Dynamo Based
http://ferd.ca/beating-the-cap-theorem-checklist.html

23. http://paul-barford.blogspot.com/2015/01/sappho-pap-obbink-further-painting-into.html
SPLIT OUTYOUR CONTROL PLANE

24. EC2
S3
RDS
Dynamo
Cloudfront
CDN
Route53
DNS
Cloudwatch
Monitoring

25. Cloudfront
CDN
Route53
DNS
Cloudwatch
Monitoring

26. Control
plane
Separate
from workload
DNS & CDN
Your best friends
Latency or
Accuracy
Pick one to sacrifice
for resilience

27. USER EXPERIENCE
My tweet got posted

28. http://mclaughlindrums.com/wp-content/uploads/2013/04/Relativity-by-Escher.jpg
ORDERED CHAOS

29. Nation’s Business, 1977

30. CHAOS DEFINED
Intentionally introducing failure into a system
with the purpose of validating resilience design.

31. http://www.cnbc.com/id/102394893

32. BREAKINGTHE SYSTEM
How Confident are you?
-Next week?
-Next month?
-After that “quick patch”

33. CHAOSVS OUTAGE
Chaos
• Controlled
• Planned
• Intentional
• Microscopic user impact
Outages
• Uncontrolled
• Unpredictable
• Unintended
• Large impact

34. Single Point of Failure
Discover - Fix -Validate

35. https://github.com/Netflix/SimianArmy
http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html
CHAOS MONKEY

36. 9am-5pm
Mon-Fri
Don’t upset
your on-call
1 Instance
Per group / per day
Detect SPOF
Intentionally

37. SLOW IS HARD
Product + Business + Engineering Decisions
https://pragprog.com/book/mnee/release-it

38. Custom
Fallback
Accuracy or latency
https://github.com/Netflix/Hystrix
Fail Silent
For optional data
Fail Fast
Keep servers healthy

39. LATENCY MONKEY
Other frameworks
http://techblog.netflix.com/2014/10/fit-failure-injection-testing.html
http://www.infoq.com/presentations/failure-as-a-service-netflixhttp://

40. HTTP 5xx
1 minute duration
10-100ms
Sleep during request
1-100%
Of requests

41. PREVENT PROPAGATION
Avoid cascading failures

42. CHAOS KONG
Because regions fail
http://techblog.netflix.com/2015/09/chaos-engineering-upgraded.html

43. GeoDNS
fallback to LatencyDNS
Proxy
Cross-Region
communication
Capacity
Cost-Benefit Decision

44. "ONCE IN A BLUE MOON"
Happens at least a few times a year....
https://whisperofangels.wordpress.com/2013/08/20/once-in-a-blue-moon/

45. TAKE AWAY
Go found chaos engineering at your company RIGHT NOW

46. Most enterprises hire people to fix things. Netflix hires
people to break things….
…we should embrace Netflix's culture of "chaos engineering"
throughout organizations of all shapes and sizes.
http://readwrite.com/2014/09/17/netflix-chaos-engineering-for-everyone

47. Q & A
http://vickicaruana.blogspot.com/2011/01/are-you-afraid-to-raise-your-hand.html
@bruce_m_wong / @jiboumans
Slides - http://www.slideshare.net/jiboumans