AWS has a large and growing portfolio of big data management and analytics services, designed to be integrated into solution architectures that meet the needs of your business. In this session, we look at analytics through the eyes of a business intelligence analyst, a data scientist, and an application developer, and we explore how to quickly leverage Amazon Redshift, Amazon QuickSight, RStudio, and Amazon Machine Learning to create powerful, yet straightforward, business solutions. Speaker: Paul Armstrong, Solutions Architect, Amazon Web Services

1. ANALYTICS ON AWS
Paul Armstrong, Solutions Architect, Amazon Web Services

2. What to expect from this session
• AWS toolkit for analytics
• Analytics stakeholders
• Amazon Redshift and Amazon QuickSight
• Anomaly Detection
• Amazon Machine Learning – Churn Prediction Example
• Q & A

3. AnalyzeStore
Amazon
Glacier
Amazon
S3
Amazon
DynamoDB
Amazon RDS,
Amazon Aurora
Big data portfolio – focus on choice
AWS Data Pipeline
Amazon
CloudSearch
Amazon EMR Amazon EC2
Amazon
Redshift
Amazon
Machine
Learning
Amazon
Elasticsearch
Service
AWS Database
Migration Services
Amazon
Kinesis
Analytics
Amazon Kinesis
Firehose
Collect
Amazon Kinesis
Streams
AWS Direct
Connect
Amazon
QuickSight
AWS Import/Export

4. AnalyzeStore
Amazon
Glacier
Amazon
S3
Amazon
DynamoDB
Amazon RDS,
Amazon Aurora
Big data portfolio – focus on choice
AWS Data Pipeline
Amazon
CloudSearch
Amazon EMR Amazon EC2
Amazon
Redshift
Amazon
Machine
Learning
Amazon
Elasticsearch
Service
Amazon
Kinesis
Analytics
Amazon Kinesis
Firehose
Collect
Amazon Kinesis
Streams
AWS Direct
Connect
Amazon
QuickSight
AWS Import/Export
AWS Database
Migration Services

5. Match toolset to right persona
• Business intelligence (BI) analyst
• Primary tool is SQL
• Historical data resides in data warehouse such as Amazon Redshift
• Data scientist
• Uses programmatic languages such as R or Python
• Application developer
• Requires API to integrate with AWS services

6. B I A N A L Y S T

7. BI analyst with existing BI tools
BI Analyst
BI tools
Amazon EC2
Amazon Redshift
Amazon QuickSight API
• Primary tool is SQL
• Data is largely structured with well known data sources
• Primary concern is fast, consistent performance
• Need to extend SQL with custom functions
BI tools
Amazon EC2
Amazon QuickSight
Amazon QuickSight

8. Amazon Redshift system architecture
Leader node
• SQL endpoint
• Stores metadata
• Coordinates query execution
Compute nodes
• Local, columnar storage
• Execute queries in parallel
• Load, backup, restore via
Amazon S3; load from
Amazon DynamoDB, Amazon EMR, or SSH
Two hardware platforms
• Optimized for data processing
• DS2: HDD; scale from 2 TB to 2 PB
• DC1: SSD; scale from 160 GB to 356 TB
10 GigE
(HPC)
JDBC/ODBC

9. New SQL functions
We add SQL functions regularly to expand Amazon Redshift’s query capabilities
Added 25+ window and aggregate functions since launch, including:
LISTAGG
[APPROXIMATE] COUNT
DROP IF EXISTS, CREATE IF NOT EXISTS
REGEXP_SUBSTR, _COUNT, _INSTR, _REPLACE
PERCENTILE_CONT, _DISC, MEDIAN
PERCENT_RANK, RATIO_TO_REPORT
We’ll continue iterating but also want to enable you to write your own
Window function examples: http://docs.aws.amazon.com/redshift/latest/dg/r_Window_function_examples.html

10. Scalar user defined functions
You can write UDFs using Python 2.7
• Syntax is largely identical to PostgreSQL UDF
• Python execution is performed in parallel
• System and network calls within UDFs are prohibited
Comes integrated with Pandas, NumPy, SciPy, DateUtil, and
Pytz analytic libraries
• Import your own libraries for even more flexibility
• Take advantage of thousands of functions available through Python
libraries to perform operations not easily expressed in SQL

11. A very fast, cloud-powered, business
intelligence service for 1/10 the cost of
traditional BI software
What is Amazon QuickSight?

12. Business
User
Business
User
Amazon
QuickSight
APIAmazon QuickSight UI
Mobile Devices Web Browsers
Partner BI Products
MetadataData PrepConnectors SuggestionsSPICE
Amazon
S3
Amazon
Kinesis
Amazon
DynamoDB
Amazon EMRAmazon
Redshift
Amazon RDSFiles Third-party

13. D A T A
S C I E N T I S T

14. Data scientist with existing toolsets
Data scientist Toolkits like SAS or
R Studio installed
with Amazon EC2
Unstructured data
Amazon S3
Structured data
Amazon Redshift
• Work with unstructured datasets
• Use existing toolsets to connect to Amazon Redshift

15. Querying Amazon Redshift with R packages
• RJDBC—Supports SQL queries
• dplyr—Uses R code for data
analysis
• RPostgreSQL—R compliant
driver or Database Interface (DBI)R User
R Studio
Amazon
EC2
Unstructured data
Amazon S3
User profile
Amazon RDS
Amazon Redshift
Connecting R with Amazon Redshift blog post: https://aws.amazon.com/blogs/big-data/connecting-r-with-amazon-redshift/

16. Querying Amazon Redshift with R packages example

17. A P P L I C A T I O N
D E V E L O P E R

18. Application developers can build smart
applications using Amazon Machine Learning
Structured data/predictions
Amazon Redshift
Generate/query
predictions
Amazon QuickSight
Application
Amazon Machine
Learning
Visualize
• All skill levels
• Amazon Machine Learning technology is accessed through APIs and SDKs
• Embed visualizations in applications

19. Resources
Amazon Redshift Getting Started Guide:
http://docs.aws.amazon.com/redshift/latest/gsg/getting-started.html
Scalar UDF Documentation: http://docs.aws.amazon.com/redshift/latest/dg/user-defined-
functions.html
Introduction to Python UDFs in Amazon Redshift:
https://blogs.aws.amazon.com/bigdata/post/Tx1IHV1G67CY53T/Introduction-to-Python-UDFs-in-
Amazon-Redshift
Connecting R with Amazon Redshift:
https://blogs.aws.amazon.com/bigdata/post/Tx1G8828SPGX3PK/Connecting-R-with-Amazon-
Redshift
Databricks Apache Spark–Amazon Redshift Tutorial: https://github.com/databricks/spark-
redshift/tree/master/tutorial
Amazon ML Getting Started Guide: https://aws.amazon.com/machine-learning/getting-started/
Amazon QuickSight: https://aws.amazon.com/quicksight/

20. Real-Time Anomaly Detection
• Ingest data from website through API Gateway and Amazon Kinesis
Streams
• Use Amazon Kinesis Analytics to produce an anomaly score for
each data point and identify trends in data
• Send users and machines notifications through Amazon SNS
Amazon API
Gateway
Amazon
Kinesis
Streams
Amazon
Kinesis
Streams
Amazon
Kinesis
Analytics
Lambda
function
Amazon
SNS
topic
email
notification
users
SMS
notification
SMS
Ingest clickstream data Detect anomalies& take action Notify users

21. Predicting Customer Churn with Amazon ML

22. Supervised Learning

23. Supervised Learning
Input Outcome

24. Supervised Learning
Input Outcome
Input
Input
Input
Outcome
Outcome
Outcome

25. Supervised Learning
Input Outcome
Input
Input
Input
Outcome
Outcome
Outcome
Supervised
Learning
known historical data
Amazon ML

26. Supervised Learning
Input Outcome
Input
Input
Input
Outcome
Outcome
Outcome
Supervised
Learning
Unseen Input Same Outcome
known historical data
Amazon ML

27. Amazon Machine Learning Service

28. Amazon Machine Learning Service

29. Amazon Machine Learning Service

30. Amazon Machine Learning Service

31. Telco Churn Dataset
• US telco customers, their cell phone plans and usage
• 21 attributes, 3333 rows:
• Customer: State, Area_Code, Phone
• Plan: Intl_Plan, VMail_Plan
• Behavior: VMail_Messages, Day_Mins, Day_Calls,
Day_Charge, Eve_Mins, Eve_Calls, Eve_Charge,
Night_Mins, Night_Calls, Night_Charge, Intl_Mins,
Intl_Calls, Intl_Charge
• Other: Account_Length, CustServ_Calls, Churn

32. Telco Churn Dataset
• US telco customers, their cell phone plans and usage
• 21 attributes, 3333 rows:
• Customer: State, Area_Code, Phone
• Plan: Intl_Plan, VMail_Plan
• Behavior: VMail_Messages, Day_Mins, Day_Calls,
Day_Charge, Eve_Mins, Eve_Calls, Eve_Charge,
Night_Mins, Night_Calls, Night_Charge, Intl_Mins,
Intl_Calls, Intl_Charge
• Other: Account_Length, CustServ_Calls, Churn

33. Telco Churn Dataset
KS, 128, 415, 382-4657, 0, 1, 25, 265.100000, 110, 45.070000, 197.400000, 99,
16.780000, 244.700000, 91, 11.010000, 10.000000, 3, 2.700000, 1, 0
OH, 107, 415, 371-7191, 0, 1, 26, 161.600000, 123, 27.470000, 195.500000, 103,
16.620000, 254.400000, 103, 11.450000, 13.700000, 3, 3.700000, 1, 0
NJ, 137, 415, 358-1921, 0, 0, 0, 243.400000, 114, 41.380000, 121.200000, 110,
10.300000, 162.600000, 104, 7.320000, 12.200000, 5, 3.290000, 0, 0
OH, 84, 408, 375-9999, 1, 0, 0, 299.400000, 71, 50.900000, 61.900000, 88, 5.260000,
196.900000, 89, 8.860000, 6.600000, 7, 1.780000, 2, 0
OK, 75, 415, 330-6626, 1, 0, 0, 166.700000, 113, 28.340000, 148.300000, 122, 12.610000,
186.900000, 121, 8.410000, 10.100000, 3, 2.730000, 3, 0
AL, 118, 510, 391-8027, 1, 0, 0, 223.400000, 98, 37.980000, 220.600000, 101, 18.750000,
203.900000, 118, 9.180000, 6.300000, 6, 1.700000, 0, 0

34. Console: Creating Datasource for Amazon ML

35. Console: Creating Datasource for Amazon ML

36. Console: Creating Datasource for Amazon ML

37. Console: Building the Amazon ML Model

38. Recipe
{ "groups": {
"NUMERIC_VARS_NORM":
"group('Intl_Charge','Night_Calls','Day_Calls','Eve_Calls','Eve_Mins','Int
l_Mins','VMail_Message','Intl_Calls','Day_Mins','Night_Mins','Day_Charge',
'Night_Charge','Eve_Charge','Account_Length')” },
"assignments": {},
"outputs": [
"ALL_BINARY",
"State",
"Area_Code",
"normalize(NUMERIC_VARS_NORM)",
"CustServ_Calls"
]
}

39. Recipe: normalize() function
Account_Length Normalized Value
128 0.808771865
107 -0.047574816
137 1.175777586
84 -0.985478323
75 -1.352484044
118 0.400987732

40. Building the Amazon ML Model

41. Cost of Errors
• Cost of Customer Churn and Acquisition (false
negative):
• foregone cashflow
• advertising costs
• POS and sign-up admin costs
• Customer Retention Cost (false + true positive)
• Discounts
• Phone upgrades
• etc

42. Financial Outcome of Applying a Model
Prior Churn Churn Cost Cost without ML
14.49% $500.00 $72.46

43. Financial Outcome of Applying a Model
Prior Churn Churn Cost Cost without ML
14.49% $500.00 $72.46
False Negative True + False Pos Retention Cost Cost with ML
4.80% 12.10% + 14.30% $100.00 $50.40

44. Financial Outcome of Applying a Model
Prior Churn Churn Cost Cost without ML
14.49% $500.00 $72.46
False Negative True + False Pos Retention Cost Cost with ML
4.80% 12.10% + 14.30% $100.00 $50.40
• Threshold 0.3  0.17
• $22.06 of savings per customer
• With 100,000 customers over $2MM
in savings with ML

45. What Next?
• https://aws.amazon.com/getting-started/projects/build-machine-
learning-model/
• https://aws.amazon.com/machine-learning/developer-resources/
• Cost Threshold Calculation
https://github.com/dbatalov/cost_based_ml
• Apache Spark on EMR https://aws.amazon.com/emr/details/spark/
• Artificial Intelligence on AWS https://aws.amazon.com/amazon-ai/
• Amazon AMIs for Deep Learning https://aws.amazon.com/amazon-
ai/amis/

46. THANK YOU!