Amazon SageMaker Ground Truth: Build High-Quality and Accurate ML Training Datasets

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Build highly accurate training datasets and reduce data
labeling costs by up to 70% using machine learning

How it works
Raw Data

How it works

Creating training data

Amazon SageMaker ground truth
Label machine learning training data easily and accurately

Driving Ease of Use

Dealing with documents
is demanding
How can we make it easy?

the 137 protein structures
The performance of the various clusterings was evalu-
MethOd Num' C'USte'S Rand mdex ated using two types of measures. The first is the average
TM~score 8 89.7% silhouette width itself, which is a measure of the clus-
ppm 9 39,396 ter compactness and separation. In general, clustering is
305C 9 895% based on the assumption that the underlying data form
compact clusters of similar characteristics. Larger aver-
R50 7 92.096
age Silhouette Width means that the result of a clustering
Traditional OCR only provides a “bag of letters”

T E X T
method. Then, the proteins were clustered using the k- medoids method with the optimal
number of clusters.
The performance of the various clusterings was evalu- ated using two types of measures.
The first is the average silhouette width itself, which is a measure of the clus- ter
compactness and separation. In general, clustering is based on the assumption that the
underlying data form compact clusters of similar characteristics. Larger average
silhouette width means that the result of a clustering algorithm consists of compact clusters
which are well separated from each other, i.e. probably close to the actual data
distribution. A small average silhouette width means e.g. that one of the clusters discovered
by the clustering algorithm could be separated in two clusters, or that some
Search
index
Amazon Textract: An organized filing cabinet of
document content

Method Num. clusters Rand index
TM-score
FPFH
3DSC
RSD
VFH
Combined silhouette weights
Combined equal weights
8
9
9
7
8
7
7
89.7%
89.3%
89.5%
92.0%
85.3%
92.2%
90.2%
Aurora
Amazon Textract: An organized filing cabinet of
document content

Graceland, Memphis
Presley, Elvis Aaron
TCB Limited
12-12-1234
TN
01 08 1935 X
901 987-6543
3765 Elvis Presley Blvd.
38116
X RCA Records
Rock n Roll Health
X
Presley, Elvis Aaron
Government forms (e.g. FDA new drug
application, financial disclosure form,
incident reporting)
Tax forms (US – e.g. W2, 1099-MISC, 990,
1040; UK – e.g. P45; Canada – e.g. T4, T5)
Amazon Textract: automatic document processing
without data entry, or writing rules

Presley, Elvis AaronN A M E
Graceland, Memphis, TNA D D R E S S
12-12-1234I D
TCB LimitedC O M P A N Y
Graceland, Memphis
Presley
TCB Limited
12-12-1234
TN
901 987-6543
3765 Elvis Presley Blvd.
38116
Elvis
Elvis.Presley@yahoo.com
Amazon Textract: automatic document processing
without data entry, or writing rules

Amazon Textract
Extract text and data from virtually any document
Eliminate
manual effort
Lower document
processing costs
K E Y F E AT U R E S
Extract data quickly and
accurately
Optical Character
Recognition (OCR)
Key-value pair
detection
Adjustable confidence
thresholds
Table
detection
Bounding box
coordinates
No ML experience
required

We have many AI services with
pre-trained models…
but there is no master algorithm for
recommendations and personalization

Music Film Products Content
Tracks
Artists
Albums
Actors
Directors
Genres
Pricing
Category
Promotions
Themes
Demographics
Breaking News

Popularity Trap
Only showing most popular
items limits the individuality of
the recommendation
Cold Starts
Relevant recommendations must be able to be
surfaced even for customers with limited history
Scale
Resonant recommendations need to scale
across thousands of products and customers
Real-Time
Personalization must work at low latency, and be
responsive to the changing intent of a customer
The image part with
relationship ID rId5 was not
found in the file.

Amazon Personalize
Real-time personalization and
recommendation service, based on the same
technology used at Amazon.com. No ML
experience required.

Activity stream
Views, signups, conversion, etc.
Inventory
Videos, products, articles, etc.
Customized
Personalization
APIDemographics (optional)
Name, age, location, etc.
1. Load data
2. Inspect data
3. Identify features
4. Select algorithms
5. Select hyperparameters
6. Train models
7. Optimize models
8. Build feature store
9. Deploy and host models
10. Create real-time caches
Amazon Personalize

Real-time Works with almost any
product or content
Responsive to changes in
intent
Automated
machine learning
Bring existing algorithms from
Amazon SageMaker
Deliver high quality
recommendations
Deep learning enabled
algorithms
Easy to Use
The image part with relationship ID rId2 was not found in the file.

Customers often ask…
“How can we tap into Amazon’s
experience in machine learning?”

Accuracy is the most important factor in forecasting
A recent study of 15 US companies revealed that 15% improvement in accuracy
leads to 3% improvement in pre-tax profit*
Under forecasting leads to
lost opportunity
Over-forecasting leads to
wasted resources
*http://demand-planning.com/2018/07/12/how-much-does-forecasting-software-cost/*

External Factors
Weather, holidays, events
and trends impact
demand, and should be
integrated into forecasts
No History Available
New products or processes with no prior data are
very difficult to forecast
Additional Variables
Traditional models rarely take into consideration
additional meta-data because it is hard to obtain
Spikey or Intermittent Data
Real-world data often exhibits irregular patterns
which causes traditional models to fail

Sales
Forecast
Sales
Time

Amazon Forecast
Accurate time-series forecasting service, based
on the same technology used at Amazon.com.
No ML Experience Required

Fulfillment by Amazon (FBA) observed a 13.9% increase after switching from traditional methods to
deep learning based ones. Today, FBA’s forecasting is used by over 2M sellers to stock Amazon’s
warehouses with optimal inventory levels and fulfill demand

Historical data
Sales, inventory, pricing, etc.
Related data
Weather, competitive promotions, etc.
1. Load data
2. Inspect data
3. Identify features
4. Select algorithms
5. Select hyperparameters
6. Train models
7. Optimize models
8. Deploy and host models
Amazon Forecast
Customized
Forecasting
API
Private

Predicts spikes
accurately!
Generates forecasts
for new items
Learns relationships
between multiple
related time-series
Incorporates external
data (holidays,
promotions, and
so on)

Custom models –
no data sharing
Forecast
any time-series
Visualize and
override forecasts
Easily export to
Oracle, SAP

• Amazon Forecast is applicable across multiple
domains
• You can set your domain using the console or
via the API
• You upload datasets with different schemas
based on the domain

There are three types of datasets in Amazon Forecast:
Item metadataTarget time-series Related time-series
Related time-
series such as
price, web-hits
etc.
Historic time-
series data of
items to forecast
Attributes of the
item such as
category, genre
and brand

Evaluate accuracy
metrics and deploy
to production.
Visualize results in
the console, or
export forecasts.
Import data from
Amazon S3 buckets.
Automatically
select algorithms
through the API or
on the console,, or
choose your own.

Easy to use, through the
console and API
Works with any historical
time-series
More accurate
forecasts that integrate
external data
Consider multiple
time-series
at once
Automatic
machine
learning
Visualize forecasts in
the console & import
results into business
apps
Evaluate model
accuracy through
the console
Schedule forecasts
and model
retraining
Bring existing
algorithms from
Amazon
SageMaker

Scaling machine learning

Last year, ML was still too complicated
1
2
3
1
2
3

Amazon SageMaker: build, train, and deploy ML
1
2
3
1
2
3

SageMaker Fabric – 4 Discrete Components

Jupyter
Notebook
interface for
exploration
Built-in, high
performance
algorithms
BUILD
One-click
training
TRAIN
Automatic
Model Tuning
(Hyperparamete
r Tuning)
Amazon SageMaker components work together

Jupyter
Notebook
interface for
exploration
Built-in, high
performance
algorithms
BUILD
One-click
training
TRAIN
Automatic
Model Tuning
(Hyperparamete
r Tuning)
Amazon SageMaker components work together
Fully managed
hosting with auto-
scaling
One-click
deployment
DEPLOY
EXECUTE
Batch
Transform

JupyterLab
• JupyterLab provides a high level of integration between notebooks,
documents, and activities:
• Drag-and-drop to reorder notebook cells and copy them between notebooks.
• Run code blocks interactively from text files (.py, .R, .md, .tex, etc.).
• Link a code console to a notebook kernel to explore code interactively without
cluttering up the notebook with temporary scratch work.
• Edit popular file formats with live preview, such as Markdown, JSON, CSV, Vega,
VegaLite, and more.
• JupyterLab is built on top of an extension system

JupyterLab Extensions

Git Repositories

AWS Marketplace for machine learning
ML algorithms and models available instantly

Over 150 models and algorithms available
Natural Language
Processing
Grammar & Parsing Text OCR Computer Vision
Named Entity
Recognition
Video Classification
Speech Recognition Text-to-Speech Speaker Identification Text Classification 3D Images Anomaly Detection
Text Generation Object Detection Regression Text Clustering Handwriting Recognition Ranking
S O M E O F T H E A V A I L A B L E A L G O R I T H M S A N D M O D E L S
S E L E C T E D V E N D O R S

Optimization is extremely complex

A deep learning model compiler that lets customers train models once, and run
them anywhere with up to 2X improvement in performance

Train once, run anywhere

Amazon SageMaker Neo
Train once, run anywhere with 2x the performance
Compiler & run-time are open source 1/10th the size of original models

Recap: updates to Amazon SageMaker
Choose your
ML algorithm
Optimize your
ML algorithm
Train and
tune model
(trial and error)
Set up and
manage
environments
for training
Deploy model
in production
Scale and manage
the production
environment

M L F R A M E W O R K S &
I N F R A S T R U C T U R E
A I S E R V I C E S
R E K O G N I T I O N
I M A G E
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E XR E K O G N I T I O N
V I D E O
Vision Speech Language Chatbots
A M A Z O N
S A G E M A K E R
B U I L D T R A I N
F O R E C A S T
Forecasting
T E X T R A C T P E R S O N A L I Z E
Recommendations
D E P L O Y
Pre-built algorithms & notebooks
Data labeling (G R O U N D T R U T H )
One-click model training & tuning
Optimization (N E O )
One-click deployment & hosting
M L S E R V I C E S
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
E C 2 P 3
& P 3 N
E C 2 C 5 F P G A s G R E E N G R A S S E L A S T I C
I N F E R E N C E
Reinforcement learningAlgorithms & models ( A W S M A R K E T P L A C E
F O R M A C H I N E L E A R N I N G )

What’s next for
machine learning?

Supervised learning
Unsupervised learning
Types of Machine LearningSOPHISTICATIONOFMLMODELS
AMOUNT OF TRAINING DATA REQUIRED

Types of Machine Learning
Supervised learning
SOPHISTICATIONOFMLMODELS

Types of Machine Learning
Reinforcement learning
(RL)
Supervised learning
SOPHISTICATIONOFMLMODELS

How does RL work?
USE CASES

New machine learning capabilities in Amazon SageMaker to build,
train and deploy with reinforcement learning

Amazon SageMaker RL
Reinforcement learning for every developer and data scientist
Broad support
for frameworks
Broad support for simulation
environments including
SimuLink and MatLab
TensorFlow, Apache
MXNet, Intel Coach, and
Ray RL support
2D & 3D physics
environments and
OpenAI Gym support
Supports Amazon Sumerian and
Amazon RoboMaker
Fully
managed
Example notebooks
and tutorials

How can we get developers rolling
with reinforcement learning
(literally?

Fully autonomous 1/18th scale race car,
driven by reinforcement learning

Introducing AWS DeepRacer
HD video camera
Dual-core Intel
processorFour-wheel drive
Dual power for
compute and drive
Accelerometer
Gyroscope

AWS DeepRacer
Fully autonomous 1/18th scale race car,
driven by reinforcement learning

Introducing
The world’s first global, autonomous racing league, open to
anyone

AWS DeepRacer League
Competitive racing league for AWS DeepRacer
Compete virtually online
Train models with RL
Race in trials
Fastest times advance
Top 10 times score
Final at re:Invent

Other ways to
getting started…

Machine Learning University
Uses the same materials
used to train Amazon
developers
Foundational knowledge
with
real-world application
Structured
courses and
specialist certification

Amazon ML Solutions Lab
Brainstorming
Custom modeling
Training
Work side-by-side with Amazon experts

Thank you!
Kris Skrinak
PSA, Global ML Segment Lead
Amazon Web Services
skrinak@amazon.com

Amazon SageMaker Ground Truth: Build High-Quality and Accurate ML Training Datasets

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Amazon SageMaker Ground Truth: Build High-Quality and Accurate ML Training Datasets

Similar to Amazon SageMaker Ground Truth: Build High-Quality and Accurate ML Training Datasets (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Amazon SageMaker Ground Truth: Build High-Quality and Accurate ML Training Datasets