The static nature of BI today results in business insight latency, that cost companies millions of dollars. Data-centric companies like web-based businesses, digital advertising, fintech and IoT need real time business detection to optimize their business performance. In this presentation, Nir Kalish, Sr. Director of Solution Engineering, explains how this can be achieved using Predictive Anomaly Detection. Presented at ODSC West, November 2016.
5. Getting Business Insights using traditional BI tools?
Monitoring Systems?
Maintenance,
not automated
False
Positive
No Real timeMillions of
metrics
%
0 1 0 1 1 0 1 0 1 0 1 0
5
10. Automatic Anomaly Detection in five Steps
Metrics
Collection –
Universal, scale
to millions
Normal
behavior
learning
Abnormal
behavior
learning
Behavioral
Topology
Learning
Feedback
Based Learning
1 2 3 4 5
10
11. Automatic Anomaly Detection in five Steps
Metrics
Collection –
Granular, scale
to millions
Normal
behavior
learning
Abnormal
behavior
learning
Behavioral
Topology
Learning
Feedback
Based Learning
1 2 3 4 5
11
12. Anomaly Detection in every granularity
Number of
Purchases
Product
Categor
y
Geo
Device
OS
Revenue
12
$ gift card
TV model
Phone model
Gift cards
Cell Phones
Electronics
US
EMEA
APJ
iOS
Windows
Android
13. Large Scale Anomaly Detection System Architecture
Kafka
Events
Queue
Anomaly
Grouping
Signals
Correlation
Map
Real-Time
Rollups Store
Cassandra
Anodotd
REST
WebApp
Online
Base Line
Learning
Aggregator
Elasticsearch
DWH S3
HADOOP/
Spark
HIVE
Offline
Learning
Management
&
Portal
Anodot-Web
User Mgmt
RDBMS
Customer DS
Agent
• 5.4 billion daily samples
• 120,000,000 metrics
• 240,000,000 models
• updated with each sample
• 500,000,000 correlation links
• Updated daily
• 14,000,000 seasonal models
• Updated daily
• 30 types of learning algorithms
• Metric classification,
seasonality detection, trend,
baseline models, clustering
algos, LSH, …
• And counting…
14. Automatic Anomaly Detection in five Steps
Metrics
Collection –
Universal, scale
to millions
Normal
behavior
learning
Abnormal
behavior
learning
Behavioral
Topology
Learning
Feedback
based learning
1 2 3 4 5
14
15. Static Thresholds versus Anomaly Based Alert
Anomaly Based Alert will find the problems hours before
the static based one
15
18. Learning the normal behavior: Not all signals are created equal
Smooth Irregular
sampling
Multi Modal Sparse
Discrete “Step”
18
Step 1
Classify signal
to Category
Step 2
Match
Category with
Baseline
Distribution
and Algorithm
30. Weekly anomaly stats: The importance of all steps
Based on 120,000,000 metrics
Normal
behavior
learning
Abnormal
behavior
learning
Behavioral
Topology
Learning
31. 31
Anomaly Detection is every where
Social
Fintech
IT Ad-Tech
E-commerce
IOT
Business
Incident
Detection
32. 32
Current Anodot Customers – Partial List
- Pedro Silva, Senior product, Credit Karma
It used to take us up to several days to identify an issue on a specific
page, offer, or service that was draining our revenues. Anodot
identifies when a metric increases or decreases in real time, so we
can resolve it quickly, before business suffers or revenue is lost.
Talking points: Ad tech tends to have KPIs that measure all the time and are smoother ---- a lot less irregular. Probably due to much heavier activity (with exchanges and ads – and end users that are a lot more).
E-commerce KPIs are driven by users in the site/mobile/app --- so tends to be more of metrics that are not “smooth”. (17% difference from ad-tech and 5% less than all companies.
Both have at least 50% metrics that are not “smooth” but require various other algos.