AWS OpsWorks for Chef Automate provides a fully managed Chef server and suite of automation tools that give you workflow automation for continuous deployment, automated testing for compliance and security, and a user interface that gives you visibility into your nodes and their status.
Learning Objectives:
• Learn about the capabilities, features and benefits of AWS OpsWorks for Chef Automate
• Learn how you can automate configuration management using AWS OpsWorks for Chef Automate
• Learn how to get started using AWS OpsWorks for Chef Automate
2. What to expect from this session?
Understand how configuration management lets you
refer to your infrastructure as code
Understand how AWS can help you use configuration
management to save time
Discover the best practices of setting up your
infrastructure, host configuration, and application
3. Background
Moving to the cloud and AWS allows you to provision and
manage infrastructure in new ways:
Scale can be achieved without complicated capacity
planning
Infrastructure can be provisioned in minutes
You are now a part of a fast moving environment that
requires constant attention
4. What is configuration management?
A practice in which code is used to define and maintain the
state of both new and existing resources throughout their
entire life cycle.
5. Why do I need configuration management?
Store your configuration information in one place
Spin up blank resources that work perfectly every time
Make changes things in a single place and propagate them
Create dev and test environments that mimic your production
6. Compute Resources
Operating System and
Host Configuration
Application Configuration
Amazon Elastic Compute
Cloud (EC2)
On-premises compute
resources (Servers)
…
Files
Directories
Networking
Symlinks
Mounts
Registry Key
Users
Groups
Packages
Filesystems
…
Application dependencies
Application configuration
Service registration
Credentials
…
7. Infrastructure needs ongoing management
Package updates?
New software?
New configurations?
New app deployments?
Environment specific changes?
Run commands across all hosts?
Be on top of all running resources?
8. Ongoing management requires proper tooling
Some common challenges:
Changing a vhost configuration on every web server across
multiple environments (dev, stage, prod)
Installing a package on certain hosts to test out newer versions
Changing LDAP config on every running Amazon EC2 Linux host
What tools can I use to tackle some of these challenges?
9.
10. What is Chef Automate?
Refer to your infrastructure as code (cookbooks & recipes)
Consistently install, configure, manage, deploy and scale
applications
Align resources with specific policies
Save time by automating manual tasks
11. How does it work?
Simple client-server
architecture
Connecting resources to a
Chef server
Resources pull
configuration updates from
the Chef server Config A Config B
12. How can you set this up?
1. Setup the Chef server with cookbooks, recipes roles.
2. Install the Chef client on the instance (or server).
3. Register the instance with the Chef server as a Chef node.
4. Assign node with a role (e.g. web server, app server, db server).
5. The Chef client pulls the recipes from Chef server (based on role).
6. The Chef server determines the applicable recipes (by role).
7. The Chef client applies the recipes on the node by doing a “Chef run”.
8. The Chef client pulls the Chef server every 30 minutes.
13. How does it look like?
The Chef client pulls
configuration updates from the
Chef server every 30 minutes.
The Chef client will only make
configuration changes when
the node is out of spec.
The Chef client can react to
changes using by using Chef
search.
14. Support for community tools
ChefDK
Knife
Chef Client
Community cookbooks and recipes
TestKitchen
kitchen
create
kitchen
converge
kitchen
login
verify
kitchen
destroy
15. Chef recipe example – configure Apache
# Install Apache and start the service.
httpd_service ‘default' do
listen_ports ['81', '82']
threadlimit '4096'
action [:create, :start]
end
# Add the site configuration.
httpd_config ‘default' do
instance ‘default'
source ‘mysite.conf.erb'
notifies :restart, 'httpd_service[default]'
end
.....
16. Chef recipe example – configure Apache
# Create the document root directory.
directory '/var/www/default/public_html' do
recursive true
end
# Write the home page.
file '/var/www/default/public_html/index.html' do
content '<html>This is a placeholder</html>'
mode '0644'
owner 'web_admin'
group 'web_admin'
end
.....
17. Chef recipe example – configure PHP
# Install the mod_php5 Apache module.
httpd_module 'php5' do
instance ‘default'
end
# Install php5-mysql.
package 'php5-mysql' do
action :install
notifies :restart, 'httpd_service[default]'
end
18. Get visibility into the state of your nodes
Visibility – A view into convergence, compliance, cookbooks, recipes and more.
19. Not only a Configuration Management tool
Workflow – A continuous delivery pipeline of infrastructure and applications.
20. Not only a Configuration Management tool
Compliance - Discovery and analysis of compliance risks across environments
22. What is AWS OpsWorks for Chef Automate?
The place you go to for configuration management on AWS
Offers a fully managed Chef Automate server
OpsWorks
23. How can I create an AWS managed Chef server?
Easy to get started, get a Chef Automate server in 10 minutes.
24. What else can I set up?
Setup a weekly maintenance window
Automatic security updates
Automatic Chef version upgrades
25. What else can I set up?
Setup a daily/weekly backup schedule
26. What else is left for me to do?
Nothing, this is a fully managed configuration management
service:
Automatic backups
Automatic security updates
Automatic Chef software updates
You can focus on writing cookbooks and recipes that meet
your needs.
27. What other benefits do I get from the service?
Automatic instance to Chef server registration
Secure and easy scaling using Auto Scaling Groups
No separate license fees, only pay for what you use
Supports both Amazon EC2 and on-prem resources
Best practices, AWS support and guidance
28. Where does it come in the tool chain?
Bootstrap instances with the right configuration
Update the configuration of running instances
Assure instances comply with a pre-defined policy
A part of your Continues Integration and Continues
Delivery pipeline
30. How do I get started?
Grab some community cookbooks
https://supermarket.chef.io/
Learn more
https://www.chef.io/automate/
Get started
https://aws.amazon.com/opsworks/
Using configuration management to Codify your infrastructure
Save time - automate tasks - package updates, setting up app dependencies, deploying app
Share best practices on – properly setup infra, host config, all to support app
Background on why we have configuration management
In pre-cloud days, infrastructure was static
Today, scale up based on actual user traffic
Getting infra in minutes does mean our jobs just got easier, on the contrary.
We expected to keep up with that speed of changes while delivering the same consistent results
We are expected to iterate quickly on dev, test and prod env with no mistakes @ problem large scale
Because our fast-moving infrastructure requires constant attention
How are you supposed to keep up with that speed while operating in scale?
One answer is configuration management.
Lots of ways, this is how most customers see it
Since the resource is created until the time it is terminated, and everything in between
Remove logs to clear some space, or remove several ssh keys from for users that left
Give some sense of direction of why you need CM
Use the following illustration
Provision compute resources somewhere… what do you do after their provisioned?
Setup the host configuration, create files, directories, network settings.
Everything I need to support application, dependencies, configuration.
I got my application running – Am I done?
Still needs ongoing management, Package updates, deploy new app version
Mitigate operational problems, for example restarting a service my a fleet of EC2 instances.
How do I get in control of my resources, especially if there is a lot of them?
Those who have been doing this for a while – tools!
What tools can I use to tackle some of these challenges?
Today talk about one of those tools - Chef Automate.
Chef latest commercial software bundle - configuration management and a lot more.
Refer to your infrastructure as code - more specific, Ruby code.
Cookbooks and recipes help you codify your Host and Application configuration.
1- install and configure software components, 2- manage files
3- deploy and scale applications, 4- execute other recipes.
Consistently scale you application whether its made of 5 resource or a 1000.
These recipes help you maintain the sate or policy I define for every resource
Accidently or maliciously changes the sate of your resource
Chef use recipes to enforce the policy - bringing the resource back to its original state
Save time by automating manual tasks
Configuration (vhost config, package install) defined in recipes – not manual login
Recipe – spend hour automating a 5-minute task that you do 100 times per week.
How does it work? Simple client server architecture
Have Chef server, and your instances.
Once instance identifies with the Chef server become Chef node.
Ask the server what sort of recipes do I need to run - As simple as that.
Two configurations (config a: web server – config b: different app or different tier, same app).
This configuration will be defined and enforced by recipes
Step 1:
Setup a Chef server, place some cookbooks recipes on it and define some roles
Where to get cookbooks/recipes soon, once you have them - you can associate them with roles.
Roles define what a certain instance supposed to do
Define a web-server role and associate 3 recipes
Chef client executes the recipes in the same order every time every 30 minutes, provides consistent, and repeatable results.
The Chef client only make configuration changes when the node is out of spec.
Exmp - Installs a particular service on a node only if the service doesn&apos;t already exist or if it&apos;s out of date.
To support system integration, harvests, and make available to recipes through Chef search
Exmp – HA proxy load balancer search for new web servers and begin to route traffic to them.
Exmp – DB server will update its access list to allow new application servers to read/write access.
How do those recipes look like?
Test Kitchen to fail fast
Could create a on-box using vagrant, EC2, even Azure
Don&apos;t have to re:ivent the wheel, chef has a supermarket
Community cookbooks and recipes, run virtually anything: mysql, nginx, postgress, php and a lot more.
Recipes can contain other recipes
Lets say we want to setup our very own web server.
Install the Apache package and start and enable its service.
And create, configure a custom site.
Installing apache and starting the service
Override default parameters: listen_ports, thread limit, timeouts and more.
The :create action handles package installation, directory setup, and other OS settings.
The :start action starts the httpd service.
On a Debian, use apt get install. RHEL or CentOS, use yam install.
Chef abstracts all of this for you.
Configuration files works across various platforms / OSs.
Additional configuration parameters, using .erb template file
Notifies command restarts the apache service
Compare the existing new config file with existing and notifies
Create the public_html folder where our website
Create a temporary html file - index.html
Placeholder content to place in that file.
Give read write permissions to the owner.
I want my website to be able to run PHP code
Install the PHP Apache module, enables Apache to interpret PHP files.
Install php5-mysql package, which enables PHP code to connect to MySQL databases.
Apache needs to be restarted to enable PHP to use the php5-mysql package.
Runs if installed
Runs if out of date
Just like this recipe, I can use many others from Chef’s supermarket.
Use recpies to scale consistently, having each resource run
Once running, every instance will use those same recipes to adapt to changes
View into the state of your cookbooks, recipes, roles and most importantly your nodes,
Whether what is defined in the recipe meets the state of the node.
Chef calls converge status.
Data warehouse platform – use through the UI to get insights into the state of your environment.
A specific package is installed on all the instances that have a web-server role
Chef is not only a configuration management tool but a lot more.
Move a change in your host or application configuration from the developer’s workstation all the way to production.
Verify changes locally, peer-review to approve, run build process, acceptance tests,
Union (integration tests),
Rehearsal (staging),
And Delivered – which releases your artifact to production.
Through the use of Chef recipes.
Has dashboards that let you track each change as it goes through the pipeline
Chef Compliance lets you generate customizable reports to identify:
Compliance issues, security risks, and even outdated software.
Aggregated in the Visibility console
Write your own compliance rules, or you can get started quickly by using built-in ones.
Use some predefined rule-sets for existing security frameworks
So now you know what Chef Automate is…
Werner’s keynote, we just launched a new service under OpsWorks in collaboration with Chef.
Opsworks - AWS service gives operational tools that help you maintain fleets of instances using Automation
Already supported configuration management in the past, which is why we consider OpsWorks, as…
You can get your very own single tenant managed Chef Automate server.
How is this different than setting your own server?
There is not setting up.
Supported regions
Server types
Fully managed service – management part.
This maintenance window happens once a week and takes up to an hour.
Non disruptive, incremental
Backups contain cookbooks recipes, roles and all the collected node information
At any point time, for any reason, you can restore, or create another copy of your Chef server.
Amount of backups, like 10 backups
We take care of the automatic backups, and the subsequent restores
And the updates and upgrades.
Focus on cookbooks recipes
Automate so you can keep up with the speed infra changes.
We support automatic instance to Chef server registration
Used Chef in the past know to use Knife
Desktop serves as mediator to handle auth and registration
This is ok for static infra
Customer that want scale have to build tools
Support for registration / de-registration APIs
Give you code snippets
Automate auth and registration
Chef commercial sofware, no separate license fees, fee for every hour
Use 5 hours, pay for 5 hours
Free-tier, up to 10 nodes
Worked with Chef, support and guidance – CM in best possible way
What would I normally use it for? Most of our customers use it for
Bootstrap instances, use community knowledge, mysql, postgress, nginx, php
Propagate a configuration update across a fleet of web servers, in 30 min all nodes update
Chef recipes to enforce policy (db server) even if changes made, Chef bring back to spec
Seen Workflow – drive a change from dev desktop to production with Chef recpies
We also have a tool of our own called AWS CodePipline.