1. Brian Henerey
Director of Technology & Operations, OpinionLab
bhenerey@gmail.com | @bhenerey | October 2014
Controlling Devops
2. Me
• Working in tech since 1998
• Discovered Puppet – 2006
• Discovered Devops – 2010
• First Devopsdays – TODAY!
3. Trust us, we’re Engineers!
• Did you delete those users accounts?
• Do those automated backups work?
• Do you know what code was released to Prod?
• Who authorized those changes?
• Who reviewed them?
• Are people ssh-ing into Prod? Did they make
any changes?
• How many people have access to customer
data?
4. Trust, Actually
• Trust is a core element of Devops
• We want to trust and empower our people, that’s
why we hired them
• Yet, we need some controls in place…
5. This Talk...
• I’m going to briefly describe what a SOC2 is and
how it works
• I’m going to tell how to rub some Devops on one
6. SOC(2) it to me
A Report on the Controls at a Service Organization
Relevant to:
– Security
– Availability
– Confidentiality
– Processing Integrity
– Privacy
These are called ‘Trust Services Principles’. Defined
here: http://en.wikipedia.org/wiki/Service_Organization_Controls
American Institute of CPAs (AICPA)
7. Why would you want one?
• To assert that your company meets all the
criteria of the Trust Services Principles
• You want an external auditor to make a
statement that you’ve done this
8. Who has SOC2 reports?
• SaaS’s
• Datacenters
• AWS (http://blogs.aws.amazon.com/security/blog/tag/SOC+2)
9. How does a SOC2 work?
You/Your Org
• Write System Description
• Write Controls
• Write Policies based on Controls
• Write Assertion about your System
Auditor
• Helps you prepare the above
• Performs audit for 3-12 months
• Produces Auditor’s report
10. What does the have to do with
Devops?
1. Accountants are more familiar with traditional IT
practices
– Devops practices require different controls
2. A SOC2 is driven by mitigating risk.
– So is DevOPs
11. What about ITIL?
• ITIL is the baby that got thrown out with the
bathwater.
• This is just my experience.
12. Over-optimized for speed
• There’s a ton more emphasis on speed of
feature delivery than creating operable systems
• You can not wish the operational pain away
• So let’s put some controls in place at the start
13. Getting started with Controls
TSP
Criteria
Risk
Example
Control
Your
Control
Evidence
14. What is a Control?
• Your organization’s practice for mitigating
specific risks
• Tip: You’re better off with multiple controls to
address each Risk
15. What kind of Controls are there?
• AICPA introduced the ‘common criteria’ which
cover:
– Organization and management
– Communications
– Risk management and design and implementation of
controls
– Monitoring of controls
– Logical and physical access of controls
– System Operations
– Change Management
16. Example Control
• Risk: “Breaches and incidents recur because
preventive measures are not implemented after a
previous event. “
• Control: “At least monthly, the Technology
Management Team meet to review the Operations
Review Register and discuss plan for resolution of
issues, or recap of resolved issues.”
17. Evidence of Control Performing Well
• “Monthly meeting minutes OR checklist of categories
of issues/projects discussed at meetings, attendees,
date, and indication if any projects/issues need to be
communicated to other employees.”
• Tip: this is what you want automated or baked into
your processes as much as possible. No one likes
busy work
18. Is a Control a Policy?
No.
• Controls are used only by the auditor
• Policies and Procedures are written for
employees with clear instructions on how to
perform their jobs.
19. How do I write a control?
Auditor
interviews me
Auditor
writes
Review
together
20. How many controls will I need?
• Around 200
– Maybe 20% of these are duplicates. The means the
same control addresses multiple risks.
• I spent 50-60 hours with the auditor writing
controls over 2-3 months
21. These are your Controls!
• Start off by writing what your company actually
does
• The most valuable part of writing these controls
is discovering areas to improve
22. What do Accountants like?
• Clarity
• Hierarchy-based Approval
• Consistency
• Ownership
• Accountability
23. How is Devops different?
• More collaboration
• More autonomy
• Less segregation of responsibilities
• Broader range of impact
– Systems, code, security, networks, databases
• More bottom-up leadership
24. Devops Criteria for writing Controls
• Lightweight
• Minimal impact to workflow
• No bureaucracy
• No busy work
• Automate/bake-in all the evidence
• Team members empowered to do their jobs
25. Up-front versus Back-end Controls
• Up-front control is more powerful, but more
stifling
– People cheat the system in favor of ‘getting things
done’
• Devops controls are on the back-end: The team
is trusted to make changes. We’ll review what
changed after the fact.
– Need multiple controls to increase the effectiveness
27. Leverage your ticketing system
• Already has history
• Already can create reports
• Easy to add approval to it
• Can automate ticket creation for
weekly/monthly/quarterly tasks
• Easy to add screenshots or attachments as
evidence
28. Leverage your version control
• Name your branches/commits based on Ticket-
ID
• Can review list of commits after deployment to
make sure changes were approved/intended
29. Leverage your tools + logging
• Have your tools create log events
– Who’s using the tool
– Datestamp
– What change was made
– i.e. Chef-handler for logstash
• https://github.com/lusis/logstash_handler
30. Side benefit of doing this
• The more you think about your organization,
you’ll probably find things you should really be
doing but aren’t.
• You’ll also find you aren’t consistent in your
behavior, which is why you need layers of
Control.
31. Example: Alert fatigue
• You probably already have a policy on
responding to alerts
• If you’re not following this process, perhaps it’s
because you have too many alerts, or they’re
un-actionable
• If it’s hard to follow your process, it’s because
you have debt in your system which should be
paid down
32. Detailed example of a Control
Common Criteria 5.6
“Logical access security measures have been
implemented to protect against security,
availability, or confidentiality threats from sources
outside the boundaries of the system.”
34. CC5.6 Control examples 1
“For VOC system infrastructure components,
logical access is controlled through each
component's native security using group/role
based permissions when possible. The Windows
network is further protected by a 3rd party multi-
factor authentication system.”
Evidence: Network diagram that includes VOC
infrastructure components
35. CC5.6 Control example 2
“For VOC custom developed applications, logical
access is controlled through role-based security
with configurations stored in the applications'
underlying database. For the 3rd party reporting
tool that is integrated with the VOC custom
developed applications, logical access is controlled
through that tool's native security.”
Evidence: Diagram or documentation that
identifies the software/applications that comprise
the VOC system
36. CC5.6 Control example 3
“Firewalls are initially setup as deny all and then
configured to allow access for approved services.
At least quarterly, the configuration of each firewall
is reviewed by the Technology Operations Team
and updated as deemed necessary according to
industry best practices.”
Evidence: Quarterly review of firewall configuration
indicating who reviewed, date reviewed, and steps
taken (e.g. firewall rules updated to…).
37. CC5.7 Control example 4
“Remote access to VOCF systems and
infrastructure components requires single factor
VPN authentication followed by a second factor
authentication at the server or network level.
Certain VOCF system component require two
factor authentication.”
Evidence: VPN configuration screen prints