All the Ops you need to know to Dev Serverless

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chris Munns – Principal Developer Advocate – AWS Serverless
All the Ops you
need to know to
Dev Serverless

About me:
Chris Munns - munns@amazon.com, @chrismunns
• Principal Developer Advocate - Serverless
• New Yorker
• Previously:
• AWS Business Development Manager – DevOps, July ’15 - Feb ‘17
• AWS Solutions Architect Nov, 2011- Dec 2014
• Formerly on operations teams @Etsy and @Meetup
• Little time at a hedge fund, Xerox and a few other startups
• Rochester Institute of Technology: Applied Networking and Systems
Administration ’05
• Internet infrastructure geek

https://secure.flickr.com/photos/mgifford/4525333972
Why are we
here today?

No servers to provision
or manage
Scales with usage
Never pay for idle Availability and fault
tolerance built in
Serverless means…

SERVICES (ANYTHING)
Changes in
data state
Requests to
endpoints
Changes in
resource state
EVENT SOURCE FUNCTION
Node.js
Python
Java
C#
Go
Serverless applications

Two common cohorts of new serverless users
Developers
who need to
learn
operations
Operations
folks who
need to learn
development

Y-Hack 2013
https://secure.flickr.com/photos/psd/4389135567/

Developers
who need to
learn
operations
Operations
folks who
need to learn
development
I’m more one
of these
folks! !

Operations
folks who
need to learn
development
Developers
who need to
learn
operations
Going to
focus more
today on
these folks

4 key operational areas
• Availability
• Networking
• Security, Governance, Auditing
• Monitoring, Metrics, Logs, Performance

InternetMobile/Web
apps
AWS
Amazon
DynamoDB
Basic Serverless API technology stack
Amazon API
Gateway
AWS Lambda
functions

Is this application available?

Then our application
gets some traﬃc...

Ok! 100% Available!

For 16 invocations for
about 1 second...

Availability is also a shared responsibility between AWS and you
• If you misconfigure API Gateway and Lambda is fine, what is your
availability of Lambda?
• If your downstream services are DDOS’d what layer’s fault is it?
• More importantly, where do you resolve it?
• If you run into concurrency limits but everything else is fine, is that
an availability issue?
• Concurrent executions is very much a soft limit!

Concurrency controls
• Concurrency is a shared pool by default
• Separate using per function concurrency settings
• Acts as reservation
• Also acts as max concurrency per function
• Especially critical for data sources like RDS
• “Kill switch” – set per function concurrency to zero

Availability means something different in serverless applications
than it does for traditional “server-full” applications:
• Availability only exists at the time in invocation and so availability
becomes a % related to total invocations vs. a time period based
metric
• The failure of downstream service(s) need to be handled/reported in
a way to potentially lead to retries/safe handling
• Retries can further confuse this (depending on invoke source)
• ie. if I fail twice and the second retry succeeds, what's my
availability?

1. Lambda directly invoked
via invoke API
SDK clients
Lambda
function
Lambda API
API provided by the Lambda service
Used by all other services that invoke
Lambda across all models
Supports sync and async
Can pass any event payload structure
you want
Client included in every SDK

Lambda networking
region
AWS Lambda VPC
Lambda
function
execution
environment

Lambda networking
region
AWS Lambda VPC
Lambda
function
execution
environment
Invocations can only come in
via the AWS Lambda API

Lambda networking
region
AWS Lambda VPC
Lambda
function
execution
environment
Today that API is available
publicly in the region Lambda
is running
Invocations can only come in
via the AWS Lambda API

Lambda networking with a customer configured VPC
region
AWS Lambda VPC Customer VPC
elastic
network
interface
Lambda
function
execution
environment

region
elastic
network
interface
Lambda
function
execution
environment
Completely managed by the
AWS Lambda team
Customer
configured/managed VPC.
Customer controls Security
Groups, Network ACLs, Route
Tables

region
elastic
network
interface
Lambda
function
execution
environment
Invocations still can only
come in via the AWS Lambda
API

region
elastic
network
interface
Lambda
function
execution
environment
Invocations still can only
come in via the AWS Lambda
API
Even with a private API
Gateway endpoint or a VPC
Endpoint provided service

Do I need to put my functions in an Amazon VPC?
Putting your functions inside of a
VPC provides little extra security
beneﬁt to your AWS Lambda
functions

Do I need a VPC?
Should my
Lambda
function be
in a VPC?
Does my function
need to access
any specific
resources in a
VPC?
Does it also need to
access resources or
services in the
public internet?
Don’t put the
function in a
VPC
Put the
function in a
private subnet
Put the
function in a
subnet with a
NAT’d route to
the internet
Yes Yes
No No

Do I need a VPC?
Should my
Lambda
function be
in a VPC?
Do I need to
restrict outbound
access from my
function to the
internet?
Don’t put the
function in a
VPC
Put the
function in a
private subnetYes
No

Basic VPC Design
Lambda
Subnets
--------->
Other
Subnets
--------->
VPC
Availability Zone A Availability Zone B
Subnet Subnet
Subnet Subnet
NAT per
<----- AZ ----->
VPC NAT
gateway
VPC NAT
gateway

© 2018, Amazon Web Services, Inc. or its Aﬃliates. All rights reserved.
Basic VPC Design
• ALWAYS conﬁgure a minimum of 2 Availability
Zones
• Give your Lambda functions their own subnets
• Give your Lambda subnets a large IP range to
handle potential scale
• If your functions need to talk to a resource on
the internet, you need a NAT!
• ENIs are a pain, we know, we’re working on it !

Securing your Serverless
Infrastructure
Photo by Markus Spiske on Unsplash

Lambda permissions model
Fine grained security controls for both
execution and invocation:
Execution policies:
• Define what AWS resources/API calls can this
function access via IAM
• Used in streaming invocations
• E.g. “Lambda function A can read from
DynamoDB table users”
Function policies:
• Used for sync and async invocations
• E.g. “Actions on bucket X can invoke Lambda
function Z"
• Resource policies allow for cross account
access

"Action": "s3:*"
makes puppies cry
Photo by Matthew Henry on Unsplash

From: https://github.com/awslabs/aws-serverless-samfarm/blob/master/api/saml.yaml
<-THIS
BECOMES THIS->
AWS SAM Templates

AWS SAM Policy Templates
MyQueueFunction:
Type: AWS::Serverless::Function
Properties:
...
Policies:
# Gives permissions to poll an SQS Queue
- SQSPollerPolicy:
queueName: !Ref MyQueue
...
MyQueue:
Type: AWS::SQS::Queue
...

SAM Policy Templates
40+ predefined
policies
All found here:
https://bit.ly/2xWycnj

IAM + Lambda best practices
• Where/when possible try to leverage
the pre-created managed policies that
exist today
• If you are doing “service:*” be REALLY
REALLY REALLY sure that’s what you
should and need to do
• Keep tight lockdown on who/what can
invoke functions

I will turn on CloudTrail, Conﬁg, and CloudTrail Data Events

Photo by Julieann Ragojo on Unsplash
BONUS: Hardcoded secrets
make fish cry

Lambda Environment Variables
• Key-value pairs that you can dynamically pass to your function
• Available via standard environment variable APIs such as process.env
for Node.js or os.environ for Python
• Can optionally be encrypted via AWS Key Management Service (KMS)
• Allows you to specify in IAM what roles have access to the keys to decrypt
the information
• Useful for creating environments per stage (i.e. dev, testing,
production)

AWS Systems Manager – Parameter Store
Centralized store to manage your
configuration data
• supports hierarchies
• plain-text or encrypted with KMS
• Can send notifications of changes
to Amazon SNS/ AWS Lambda
• Can be secured with IAM
• Calls recorded in CloudTrail
• Can be tagged
• Available via API/SDK
Useful for: centralized environment
variables, secrets control, feature
flags
from __future__ import print_function
import json
import boto3
ssm = boto3.client('ssm', 'us-east-1')
def get_parameters():
response = ssm.get_parameters(
Names=['LambdaSecureString'],WithDe
cryption=True
)
for parameter in
response['Parameters']:
return parameter['Value']
def lambda_handler(event, context):
value = get_parameters()
print("value1 = " + value)
return value # Echo back the first key
value

AWS Systems Manager – Parameter Store
Centralized store to manage your
configuration data
• supports hierarchies
• plain-text or encrypted with KMS
• Can send notifications of changes
to Amazon SNS/ AWS Lambda
• Can be secured with IAM
• Calls recorded in CloudTrail
• Can be tagged
• Available via API/SDK
Useful for: centralized environment
variables, secrets control, feature
flags
from __future__ import print_function
import json
import boto3
ssm = boto3.client('ssm', 'us-east-1')
def get_parameters():
response = ssm.get_parameters(
Names=['LambdaSecureString'],WithDe
cryption=True
)
for parameter in
response['Parameters']:
return parameter['Value']
def lambda_handler(event, context):
value = get_parameters()
print("value1 = " + value)
return value # Echo back the first key
value
#somuchawesome

https://secure.flickr.com/photos/ocarchives/5333790414
Fun with logs
and metrics
Weee
eeeee

Metrics and logging are a universal right!
CloudWatch Metrics:
• 7 Built in metrics for Lambda
• Invocation Count, Invocation duration, Invocation
errors, Throttled Invocation, Iterator Age, DLQ
Errors, Concurrency
• Can call “put-metric-data” from your function code
for custom metrics
• 7 Built in metrics for API-Gateway
• API Calls Count, Latency, 4XXs, 5XXs, Integration
Latency, Cache Hit Count, Cache Miss Count
• Error and Cache metrics support averages and
percen,les

Metrics and logging are a universal right!
CloudWatch Logs:
• API Gateway Logging
• 2 Levels of logging, ERROR and INFO
• Optionally log method request/body content
• Set globally in stage, or override per method
• Lambda Logging
• Logging directly from your code with your language’s
equivalent of console.log()
• Basic request information included
• Log Pivots
• Build metrics based on log filters
• Jump to logs that generated metrics
• Export logs to AWS ElastiCache or S3
• Explore with Kibana or Athena/QuickSight

https://secure.flickr.com/photos/joeross/6544781203

Dashboards
https://secure.flickr.com/photos/joeross/6544781203

Dashboarding tips
Make all metrics available – Good news, CloudWatch
makes it easy!
Focus main landing/“on tv” dashboards on core
user/business driven metrics
• “If this metric goes up, does it directly correlate with a user
having a problem?”
Make as many metrics available across team/function as
possible
• You can now embed CloudWatch “snapshots” in emails and
other places!

Dashboarding tips
Make all metrics available – Good news, CloudWatch
makes it easy!
Focus main landing/“on tv” dashboards on core
user/business driven metrics
• “If this metric goes up, does it directly correlate with a user
having a problem?”
• Make as many metrics available across team/function
as possible
• You can now embed CloudWatch “snapshots” in emails and
other places!

Tweak your function’s computer power
Lambda exposes only a memory control, with the % of CPU
core and network capacity allocated to a function
proportionally
Is your code CPU, Network or memory-bound? If so, it could
be cheaper to choose more memory.

Smart resource allocation
Match resource allocation (up to 3 GB!) to logic
Stats for Lambda function that calculates 1000 times all prime numbers
<= 1000000
128 MB 11.722965sec $0.024628
256 MB 6.678945sec $0.028035
512 MB 3.194954sec $0.026830
1024 MB 1.465984sec $0.024638
Green==Best Red==Worst

Smart resource allocation
Match resource allocation (up to 3 GB!) to logic
Stats for Lambda function that calculates 1000 times all prime numbers
<= 1000000
128 MB 11.722965sec $0.024628
256 MB 6.678945sec $0.028035
512 MB 3.194954sec $0.026830
1024 MB 1.465984sec $0.024638
Green==Best Red==Worst
+$0.00001-10.256981sec

Impact of a memory change
50% increase
in memory
95th percentile
changes from
3s to 2.1s
https://blog.newrelic.com/2017/06/20/lambda-functions-xray-traces-custom-serverless-metrics/

Multithreading? Maybe!
• <1.8GB is still single core
• CPU bound workloads won’t see gains – processes share same
resources
• >1.8GB is multi core
• CPU bound workloads will gains, but need to multi thread
• I/O bound workloads WILL likely see gains
• e.g. parallel calculations to return

AWS X-Ray Integration with Serverless
• API Gateway inserts a tracing
header into HTTP calls as well
as reports data back to X-Ray
itself
• Lambda instruments
incoming requests for all
supported languages and can
capture calls made in code
var AWSXRay = require(‘aws-xray-sdk-core‘);
AWSXRay.middleware.setSamplingRules(‘sampling-rules.json’);
var AWS = AWSXRay.captureAWS(require(‘aws-sdk’));
S3Client = AWS.S3();

X-Ray Trace Example

How do I figure out what’s wrong?
These tools are here, so use them!
1. Turn on X-Ray now
1. look at wrapping your own calls with it via the X-Ray SDKs
2. Don’t underestimate the power of logging in Lambda
1. Simple “debug: in functionX” statements work great and
are easy to find in CloudWatch Logs
3. The most valuable metrics are the ones closest to your
customer/use-case
1. How many gizmos did this function call/create/process/etc

FIN/ACK
4 key operational areas:
• Availability
• Networking
• Security, Governance, Auditing
• Monitoring, Metrics, Logs, Performance

Chris Munns
munns@amazon.com
@chrismunnshttps://www.flickr.com/photos/theredproject/3302110152/

?https://secure.flickr.com/photos/dullhunk/202872717/

All the Ops you need to know to Dev Serverless

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie All the Ops you need to know to Dev Serverless

Ähnlich wie All the Ops you need to know to Dev Serverless (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

All the Ops you need to know to Dev Serverless