Amazon Elastic MapReduce (Amazon EMR) is a web service that allows you to easily and securely provision and manage your Hadoop clusters. In this talk, we will introduce you to Amazon EMR design patterns, such as using various data stores like Amazon S3, how to take advantage of both transient and active clusters, and how to work with other Amazon EMR architectural patterns. We will dive deep on how to dynamically scale your cluster and address the ways you can fine-tune your cluster. We will discuss bootstrapping Hadoop applications from our partner ecosystem that you can use natively with Amazon EMR. Lastly, we will share best practices on how to keep your Amazon EMR cluster cost-effective.

1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Deep Dive: Amazon EMR
Matt Yanchyshyn - Sr. Manager, Solutions Architecture

2. Why Amazon EMR?
easy to use
launch a cluster in minutes
low cost
pay an hourly rate
elastic
easily add or remove capacity
reliable
spend less time monitoring
secure
managed firewalls
flexible
you control the cluster

3. Easy to deploy
AWS Management Console Command line
or use the EMR API with your favorite SDK

4. Easy to monitor and debug
Monitor Debug
integrated with Amazon CloudWatch
monitor cluster, node, and IO

5. Try different configurations to find your optimal architecture
CPU
c3 family
cc1.4xlarge
cc2.8xlarge
Memory
m2 family
r3 family
Disk/IO
d2 family
i2 family
General
m1 family
m3 family
Choose your instance types
Batch Machine Spark and Large
process learning interactive HDFS

6. Easy to add and remove compute
capacity on your cluster
Match compute
demands with
cluster sizing
Resizable clusters

7. Spot for
task nodes
Up to 90%
off EC2
on-demand
pricing
On-demand for
core nodes
Standard
Amazon EC2
pricing for
on-demand
capacity
Easy to use Spot Instances
Meet SLA at predictable cost Exceed SLA at lower cost

8. Read data directly into Hive,
Pig, streaming and cascading
from Amazon Kinesis streams
No intermediate data
persistence required
Simple way to introduce real-time sources into
batch-oriented systems
Multi-application support and automatic
checkpointing
Amazon EMR Integration with Amazon Kinesis

9. The Hadoop ecosystem can run in Amazon EMR

10. Bootstrap Actions
Use bootstrap actions to install applications or to configure
Hadoop
--bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop
--keyword-config-file – merge values in new config to existing
--keyword-key-value – override values provided
Configuration file
name
Configuration file
keyword
File name shortcut
Key-value pair
shortcut
core-site.xml core C c
hdfs-site.xml hdfs H h
mapred-site.xml mapred M m
yarn-site.xml yarn Y y

11. Hue
Amazon S3 and HDFS

12. Hue
Query Editor

13. Hue
Job Browser

14. Leverage Amazon S3 with EMRFS

15. Amazon S3 as your persistent data store
• Separate compute and storage
• Resize and shut down Amazon
EMR clusters with no data loss
• Point multiple Amazon EMR
clusters at same data in Amazon
S3
EMR
EMR
Amazon
S3

16. EMRFS makes it easier to use Amazon S3
• Read-after-write consistency
• Very fast list operations
• Error-handling options
• Support for Amazon S3 encryption
• Transparent to applications: s3://

17. EMRFS client-side encryption
Amazon S3
AmazonS3encryptionclients
EMRFSenabledfor
AmazonS3client-sideencryption
Key vendor (AWS KMS or your custom key vendor)
(client-side encrypted objects)

18. HDFS is still there if you need it
• Iterative workloads
– If you’re processing the same dataset more than
once
• Disk I/O-intensive workloads
• Persist data on Amazon S3 and use S3DistCp to
copy to/from HDFS for processing

19. Amazon EMR – Design Patterns

20. EMR example #1: Batch processing
GB of logs pushed to
S3 hourly
Daily EMR cluster
using Hive to process
data
Input and output
stored in S3
250 Amazon EMR jobs per day, processing 30 TB of data
http://aws.amazon.com/solutions/case-studies/yelp/

21. EMR example #2: Long-running cluster
Data pushed to S3 Daily EMR cluster
ETL data into
database
24/7 EMR cluster running
HBase holds last two years of
data
Front-end service uses
HBase cluster to power
dashboard with high
concurrency

22. TBs of logs sent
daily
Logs stored in
Amazon S3
Hive metastore
on Amazon EMR
EMR example #3: Interactive query
Interactive query using Presto on multi-petabyte warehouse
http://nflx.it/1dO7Pnt

23. EMR example #4: Streaming-data processing
TBs of logs sent
daily
Logs stored in
Amazon Kinesis
Amazon Kinesis
Client Library
AWS Lambda
Amazon EMR
Amazon EC2

24. Optimizations for Storage

25. File formats
• Row-oriented
– text files
– sequence files
• writable object
– Avro data files
• described by schema
• Columnar format
– Object Record Columnar (ORC)
– Parquet
logical table
row-oriented
column-oriented

26. Choosing the right file format
• Processing and query tools
– Hive, Impala, and Presto
• Evolution of schema
– Avro for schema and Presto for storage
• File format “splittability”
– Avoid JSON/XML files. Use them as records.
• Compression - block or file

27. File sizes
• Avoid small files
– Anything smaller than 100 MB
• Each mapper is a single JVM
– CPU time is required to spawn JVMs/mappers
• Fewer files, matching closely to block size
– fewer calls to S3
– fewer network/HDFS requests

28. Dealing with small files
• Reduce HDFS block size, e.g. 1 MB (default is 128 MB)
– --bootstrap-action s3://elasticmapreduce/bootstrap-
actions/configure-hadoop --args “-m,dfs.block.size=1048576”
• Better: use S3DistCP to combine smaller files together
– S3DistCP takes a pattern and target path to combine smaller
input files to larger ones
– Supply a target size and compression codec

29. Compression
• Always compress data files on Amazon S3
– reduces network traffic between Amazon S3 and
Amazon EMR
– speeds up your job
• Compress mappers and reducer output
Amazon EMR compresses inter-node traffic with LZO with
Hadoop 1 and Snappy with Hadoop 2

30. Choosing the right compression
• Time-sensitive, faster compressions are a better choice
• Large amount of data, use space-efficient compressions
• Combined workload, use gzip
Algorithm Splittable? Compression ratio
Compress and
decompress speed
Gzip (DEFLATE) No High Medium
bzip2 Yes Very high Slow
LZO Yes Low Fast
Snappy No Low Very fast

31. The Nielsen Company and Amazon EMR

32. Copyright©2012TheNielsenCompany.Confidentialandproprietary.
SOCIAL TV IS A CONSUMER PHENOMENON
Viewers interacting around TV programming through social to connect
with friends, fans, stars, and advertisers in real time.
of smartphone/tablet owners
use devices as second screens
while watching TV
1 Billion84percent
Tweets about U.S.
TV in 2014, sent by
25million people

33. Copyright©2012TheNielsenCompany.Confidentialandproprietary.
WHAT IS NIELSEN SOCIAL?
The leading provider of social TV measurement, analytics, and audience engagement
solutions.
• 90+ network, agency and
advertiser clients in the U.S.
• Exclusive provider of
Nielsen Twitter TV Ratings
• International presence in
Italy, Australia, and Mexico

34. Copyright©2012TheNielsenCompany.Confidentialandproprietary.
HOW DOES NIELSEN SOCIAL WORK?
Nielsen Social captures Twitter activity about every TV program across 250+ U.S.
networks, 1900+ brands, movies, and sports.
Twitter provides impressions and demographics for every tweet about TV,
which we aggregate and de-duplicate to produce Nielsen Twitter TV Ratings.

35. Copyright©2012TheNielsenCompany.Confidentialandproprietary.
NIELSEN SOCIAL – AMAZON EMR JOB FLOW
1) Country Segmentation Cluster
• Input: Global Firehose (500M TPD, 200 GB gz
files on S3)
• Output: US/IT/AU/MX Tweets (150M TPD)
• Data Volatility: Low (+/- 20%)
2) TV Tweet Matching Cluster
• Input: US/IT/AU/MX Tweets (150M TPD)
• Output: TV Tweets (2 to 25M)
• Data Volatility: High (+/- 1200%)
3) TV Analytics Cluster
• Input: TV Tweets (2 to 25M TPD)
• Output: Analytics and Reports
• Data Volatility: High (+/- 1200%)
Amazon EMR Config:
• Multiple Transient Clusters
• On Demand Instances
+ more for spikes
• Type: m4.2xlarge / m3.xl
• Job Freq: daily
Amazon EMR Config:
• Dedicated (24/7)
• Reserved Instances
• Type: m4.2xlarge
• Job Freq: every 10 min
Amazon EMR Config:
• Transient
• Reserved Instances
+ On Demand for spikes
• Type: m2.2xlarge
• Job Freq: hourly/overlap
TWITTER FIREHOSE
TV SCHEDULES
DEMOS & IMPRESSIONS

36. Copyright©2012TheNielsenCompany.Confidentialandproprietary.
TWITTER FIREHOSE
NIELSEN SOCIAL – AMAZON EMR JOB FLOW
1) Country Segmentation Cluster
• Input: Global Firehose (500M TPD, 200 GB gz
files on S3)
• Output: US/IT/AU/MX Tweets (150M TPD)
• Data Volatility: Low (+/- 20%)
2) TV Tweet Matching Cluster
• Input: US/IT/AU/MX Tweets (150M TPD)
• Output: TV Tweets (2 to 25M)
• Data Volatility: High (+/- 1200%)
3) TV Analytics Cluster
• Input: TV Tweets (2 to 25M TPD)
• Output: Analytics and Reports
• Data Volatility: High (+/- 1200%)
TV SCHEDULES
DEMOS & IMPRESSIONS
Amazon EMR Config:
• Dedicated (24/7)
• Reserved Instances
• Type: m4.2xlarge
• Job Freq: every 10 min
Amazon EMR Config:
• Transient
• Reserved Instances
+ On Demand for spikes
• Type: m2.2xlarge
• Job Freq: hourly/overlap
NIELSEN
TWITTER
TV RATINGS

37. Takeaway

38. Cost-saving tips for Amazon EMR
• Use Amazon S3 as your persistent data store
• Only pay for compute when you need it
• Use Amazon EC2 Spot instances to save >80%
• Use Amazon EC2 Reserved instances for steady workloads
• Use CloudWatch alerts to notify you if a cluster is
underutilized, then shut it down (for example, 0 mappers
running for >N hours)
• Contact your AWS sales for pricing options if you are
spending >$10K/mo on Amazon EMR

39. CHICAGO