Find out more about the most common Big Data Architectural Patterns and Best Practices on AWS

1. Francesco Rinaudo,
Sr. Manager, Solutions Architecture, AWS
May 23, 2017
Big Data Architectural Patterns
and Best Practices on AWS

2. Big Data Evolution
Batch
processing
Stream
processing
Machine
learning

3. Architectural Principles
Build decoupled systems
• Data → Store → Process → Store → Analyze → Answers
Use the right tool for the job
• Data structure, latency, throughput, access patterns
Leverage AWS managed services
• Scalable/elastic, available, reliable, secure, no/low admin
Use log-centric design patterns
• Immutable logs, materialized views
Be cost-conscious
• Big data ≠ big cost

4. Data and Analytics Flow
COLLECT STORE PROCESS/
ANALYZE
CONSUME

5. What Is the Temperature of Your Data ?

6. AWS Big Data Services
Orchestration/Transform
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
Batch
ETL/ELT
Realtime
ETL/ELT
Transactional
/ CDC
B.I. Tools
Data Science
Notebooks
Bulk Transport
File/Object Upload
Streaming Ingest
Commits Transactional
NoSQL
Data Lake
Streaming Storage
Dashboards
Batch Analytics
Interactive
Querying
Machine Learning/
Deep Learning
Realtime Analytics
…

7. AWS Big Data Services
Orchestration/Transform
EMR EC2
S3
RedshiftDynamoDB
AWS DMS (CDC)
AWS Lambda
Kinesis Analytics Amazon Athena
Amazon
QuickSight
Aurora
AWS Glue AWS Step
Functions
Kinesis
Streams
Ingest/ Collect Store Analyze/ Process
Visualization/
Consume
AWS
Snowball
ISV
Connectors
Kinesis
Firehose
S3 Transfer
Acceleration
Amazon
ElasticSearch
Reference architecture

8. Anti-Pattern
DataTier

9. Key Services Overview

10. Big Data Storage for Virtually All AWS Services
Amazon S3
• Store anything
• Object storage
• Scalable
• 99.999999999% durability
• Extremely low cost

11. Amazon
DynamoDB
Fast & Flexible NoSQL Database Service
• NoSQL Database
• Seamless scalability
• Zero admin
• Single digit millisecond latency

12. Amazon
Kinesis
Real-time Streaming Platform
• Streams, Firehose, Analytics
• Real-time processing
• High throughput; elastic
• Easy to use
• Integration with S3, EMR,
Redshift, DynamoDB

13. Amazon Kinesis
Streams
• For Technical Developers
• Build your own custom
applications that process
or analyze streaming
data
Amazon Kinesis
Firehose
• For all developers, data
scientists
• Easily load massive
volumes of streaming data
into S3, Amazon Redshift
and Amazon Elasticsearch
Amazon Kinesis
Analytics
• For all developers, data
scientists
• Easily analyze data
streams using standard
SQL queries
Amazon Kinesis: Streaming Data Made Easy
Services make it easy to capture, deliver and process streams on AWS

14. AWS Lambda
• Run your code in the cloud - fully
managed and highly-available
• Triggered through API or state
changes in your setup
• Scales automatically to match the
incoming event rate
• Node.js (JavaScript), Python, Java,
and C#
• Charged per 100ms execution time
Serverless Compute

15. Amazon
Athena
Interactive Query Service
• Query directly from Amazon S3
• Use ANSI SQL
• Serverless
• Multiple Data Formats
• Pay per query

16. AWS Glue
Fully Managed ETL Service
• Catalog data sources
• Identify data formats & data types
• Error Handling
• Manage and scale resources
• Generate ETL code
• Schedules, executes ETL jobs

17. •Fast and cloud-powered
•Easy to use, no infrastructure to
manage
•Scales to 100s of thousands of
users
•Quick calculations with SPICE
•1/10th the cost of legacy BI software
Business Intelligence
Amazon
QuickSight

18. Design Patterns

19. Amazon EMR
Real-time Analytics
Amazon
Kinesis
KCL app
AWS Lambda
Spark
Streaming
Amazon
SNS
Amazon
ML
Notifications
Amazon
ElastiCache
(Redis)
Amazon
DynamoDB
Amazon
RDS
Amazon
ES
Alerts
App state
Real-time prediction
KPI
process
store
Amazon Kinesis
Analytics
Amazon
S3
Log
Amazon
KinesisFan out

20. Interactive
&
Batch
Analytics
Amazon S3
Amazon EMR
Hive
Pig
Spark
Amazon
ML
process
store
Consume
Amazon Redshift
Amazon EMR
Presto
Spark
Batch
Interactive
Batch prediction
Real-time prediction
Amazon
Kinesis
Firehose
Amazon Athena
Files
Amazon Kinesis
Analytics

21. Interactive
&
Batch
Amazon S3
Amazon
Redshift
Amazon EMR
Presto
Hive
Pig
Spark
Amazon
ElastiCache
Amazon
DynamoDB
Amazon
RDS
Amazon
ES
AWS Lambda
Storm
Spark Streaming
on Amazon EMR
Applications
Amazon
Kinesis
App state
KCL
Amazon
ML
Real-time
Amazon
DynamoDB
Amazon
RDS
Change Data
Capture
Transactions
Stream
Files
Data Lake
Amazon Kinesis
Analytics
Amazon
Athena
Amazon
Kinesis
Firehose

22. Summary
Build decoupled systems
• Data → Store → Process → Store → Analyze → Answers
Use the right tool for the job
• Data structure, latency, throughput, access patterns
Leverage AWS managed services
• Scalable/elastic, available, reliable, secure, no/low admin
Use log-centric design patterns
• Immutable log, batch, interactive & real-time views
Be cost-conscious
• Big data ≠ big cost

23. Thank you!