From DevOpsDays Austin 2012. Video: http://www.ustream.tv/recorded/21568549

1. DevOpsSec
Applying DevOps Principles to Security
Nick Galbreath nickg@etsy.com @ngalbreath
DevOpsDays, Austin Texas, April 3, 2012
http://client9.com/20120403 nickg@client9.com

2. Slides! Video!
• Originally presented on April 3, 2012
• Latest Slides! Streaming Video!
http://client9.com/20120403
• Related interview:
http://youtu.be/Afd0u5DGxr8
• Original video stream:
http://www.ustream.tv/recorded/21568549

3. whoami
• Development background
• Lots o’ startups, book, patents,blahblahblah
• Director of Engineering at Etsy covering
• Security, Fraud, Biz Analytics, Email Infra,
Internal Systems, and everything else not
www.etsy.com “Enterprise”
• Second time working with Allspaw!
• “Oh you mean there is a name for this?”

4. Context
My biases for this talk is (Web) Application
Security, not classic Network Security or IT
Security.

5. Double-click to edit
• Double-click to edit
uhhhhhh....

6. Uhhhhhhh
What are
“DevOps Principles”

7. Blah blah blah
• Decentralization
• Shared Resources
• Risk based management
• Catholic vs protestant methologies
• Whitelist vs. blacklist mentality
• Transparency.

8. Trust But Verify

9. ...with the
acknowledgement that
• We are working in a complex system
• And in complex systems failure happens
• And failure can happen when
everyone does nothing wrong
• And given this, how can one increase
reward and reduce risk for the business

10. What does this mean for....
People?
Processes (workflow)?
Machines?

11. An Only Slightly
Contrived Example
• I trust MCR to run our network
• I can verify this by looking at our dataporn
• He trusts me that when things go wrong,
the graphs won’t be used to burn him.
• He can verify this by... seeing our Post
Mortems in action (they are open at Etsy)

12. Uhhhh....
Why DevOpsSec and
not DevOpsFoo?

13. Squeezed from Both Sides
Unreviewed Code going out,
Untrusted Data coming in
DATA UGH CODE
Makes stability and responsibility
“complicated”, even more so if there are
walls between groups.

14. Latent Problems
There are operational problems right now
just not manifested.
There are security problems right now just
not exploited.

15. Cultural Problems
• Both have severe failure causes
• Both Ops and Security have a “say no”
perception
• “Operations” and “Security” are services
groups but frequently not viewed as such

16. Ok, back to the
regularly scheduled
programming

17. DevOpsSec
E 2
Applying DevOps Principles to Security
A K
Nick Galbreath nickg@etsy.com @ngalbreath
DevOpsDays, Austin Texas, April 3, 2012
http://client9.com/20120403 nickg@client9.com

18. MTTR
Mean Time To Resolve

19. SHIT HAPPENS
Security problems will occur

20. How Fast Can You
Deploy or Rebuild
• Your Firewall,VPN, Load Balancer
• Your Operating System, Critical Servers
• Your Database, server, schema, data
• Your Application, patches
• Any other configuration file
in a consistent, sane manner

21. Being able to deploy
quickly is my #1
security feature
This implies a standardized, automated
system and configuration management.

22. I Call Bullshit
Doesn’t the rapid rate of
change in a continuous
integration environment
mean things are less secure?
Well compare this to....

23. We’ll rush that security fix.
It will go out in next release
in about 6 weeks.
former vendor at Etsy

24. MTTD
Mean Time To Detect

25. It’s ok if we have a few extra
firemen waiting around in
case there is a fire
I’m more concerned we
won’t know there is a
fire until the house is
burnt down
Conversation between Chad Dickerson and Nick Galbreath, Etsy 2011

26. Segmentation Faults
• Why is your server falling over?
• From the same IP address.
• Over and over
Maybe time to patch? Also check
out your server 500 errors.

27. Database Syntax Errors
Almost game over here.
Whose is responsible for these anyways?
Demand zero-tolerance for database syntax errors.

28. SQLi Attacks
The shittest check that works.
Will undercount by at least 10x
ProTip #1: using regexp for sqli is fail
ProTip #2: write a unit test for this.

29. And Graph It
Security is no longer a binary event.

30. Got that?
Security is not a
binary event.
You are being
attacked constantly.
THIS IS AWESOME

31. Attack Driven Testing
Security testing in dev is hard.
Use actual attacks/probes to guide testing.
• Server 500 errors
• Core Dumps Can you
automate
• CSRF Failures verification?
• XSS Attempts
• Login Failures
• Password Resets

32. TESTING
If infrastructure is code, then
doesn’t it need testing too?

33. assertyour production
environment
Who knew that
writing solid C code is
similar to running a
complex system.
Writing Solid Code
Steve Maguire
1993

34. assert this
• This page is always SSL
• This page requires sign-in
• This page never is publicly available
• Google never crawls this page
• This page is not being routed by the CDN
• This port is never open

35. Reuse your unit test
framework to test
production config

36. Reuse your
continuous integration
environment
ClamAV. Yes, we antivirus
our source code.

37. Other Topics in
DevOpsSec

38. Post Mortems
• All security issues are “P1” or “P2” (fix
now, or fix by end of week)
• Even for internal applications.
• All get a post-mortem
• Great educational experience and
knowledge transfer

39. Hiring
• Strict no asshole policy.
• Security is a services business.
• “Product Security” is in-house consultancy.
• Can you take people who are interested
and train them? TBD.
http://www.sans.org/
https://www.owasp.org/

40. Extend the Perimeter
• Working on a training program for both in-
house employees, contractors, and external
vendors.
• “Device Tune-Up Day” -- employees bring
their home computers in for a tune-up.

41. Nick Galbreath nickg@etsy.com @ngalbreath
DevOpsDays Austin Tx 2012
http://client9.com/20120403 nickg@client9.com