SlideShare ist ein Scribd-Unternehmen logo
1 von 29
Downloaden Sie, um offline zu lesen
Raven: End-to-end Optimization
of ML Prediction Queries
Konstantinos Karanasos, Kwanghyun Park
Gray Systems Lab, Microsoft
App
logic
offline
online
Model Inference
Featurization Model
Model
optimization
policies
orchestratio
n
Data
Catalogs
Governance
Model
Tracking
& Provenance
Access
Control
Logs &
Telemetry
policies
Decisions
Live Data
deployment
other data
featurizatio
n
Model
Training
Model Development / Training
offline feat.
Model
Enterprise-grade ML lifecycle
Data Scientist
Analyst/Developer
model
training
model scoring
data exploration/
preparation
data selection/
transformation
model
deployment
Use Case: Length-of-stay in Hospital
Model:
“Predict length of stay of
a patient in the hospital”
Prediction query:
“Find pregnant patients that
are expected to stay in the
hospital more than a week”
Featurization Model
Container
REST
Prediction Queries: Baseline Approach
policies
HTTP
WebServer
App logic
ODBC
DBMS
Enterprise Features
• Security: data and models outside of the DB
• Extra infrastructure
• High TCO
• Lack tooling/best-practices
Performance
• Data movement
• Latency
• Throughput on batch-scoring
Prediction Queries: In-Engine Evaluation
policies
HTTP
WebServer
App logic
ODBC
DBMS
Enterprise Features
• Security: Data and models within the DBMS
• Reuse Existing infrastructure
• Language/tools/best practices
• Low TCO
Performance ?
• Up to 13x faster on Spark
• Up to 330x faster on SQL Server
Raven: An Optimizer for Prediction Queries in Azure Data
+
data
models
Unified IR
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
MQ: inference query
Runtime
Code gen
+
Optimized
IR
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
MQ: inference query
Runtime
Code gen
+
Embed high-performance
ML inference runtimes
within our data engines
Express data and
ML operations in
a common graph
Constructing the IR
Raven IR operators
Relational algebra
Linear algebra
Other ML operators and data featurizers
UDFs
Static analysis of the prediction query
Support for SQL+ML
Adding support for Python
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
MQ: inference query
ML Inference in Azure Data Engines
SQL Server
PREDICT statement in SQL Server
Embedded ONNX Runtime in the engine
Available in Azure SQL Edge and SQL DW
(part of Azure Synapse Analytics)
Spark
Introduced a new PREDICT operator
Similar syntax to SQL Server
Support for different types of models
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
pa
Un
Static
Analysis
MQ: inference query
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
MQ: inference query
Runtime
Code gen
+
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
MQ: inference query
Runtime
Code gen
+
Q: “Find pregnant patients
expected to stay in the hospital
more than a week”
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
MQ: inference query
Runtime
Code gen
+
Raven: An Optimizer for Prediction Queries
+
+
Runtime
Code
Gen
Raven optimizations in practice
(name, model) AS
”,
eline import Pipeline
rocessing import StandardScaler
import DecisionTreeClassifier
n’, FeatureUnion(…
scaler’,StandardScaler()), …))
reeClassifier())])”);
Data Scientist)
ng model (Data Analyst)
rbinary(max) = (
OM scoring_models
e = ”duration_of_stay“ );
nfo AS pi
ts AS be ON pi.id = be.id
tests AS pt ON be.id = pt.id
ngth_of_stay
L=@model, DATA=data AS d)
ay Pred float) AS p
= 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
Runti
Code
1. Predicate-based
model pruning
2. Model projection
pushdown
3. Model splitting
4. Model-to-SQL
translation
5. NN translation
6. Standard DB
optimizations
7. Compiler
optimizations
1. Avoid unnecessary computation
Information passing between model and data
2. Pick the right runtime for each operation
Translation between data and ML operations
3. Hardware acceleration
Translation to tensor computations
(Hummingbird)
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
MQ: inference query
Runtime
Code gen
Raven optimizations: Key Ideas
0
1,000
2,000
3,000
4,000
5,000
DT-depth5 DT-depth8 LR-.001 GB-20est DT-depth5 DT-depth8 LR-.001 GB-20est
Hospital - 2 billion rows Expedia - 500 million rows
Elapsed
time
(seconds)
End-to-end inferene query time
SparkML Sklearn ONNX runtime Raven
Performance Evaluation: Raven in Spark (HDI)
Best of Raven:
• Decision Trees (DT) and Logistic Regressions (LR): Model Projection Pushdown + ML-to-SQL
• Gradient Boost (GB): Model Projection Pushdown
SELECT PREDICT(model, col1, …)
FROM Hospital
SELECT PREDICT(model, S.col1, …)
FROM listings S, hotels R1, searches R2
WHERE S.prop_id = R1.prop_id AND S.srch_id = R2.srch_id
Raven outperforms other ML runtimes (SparkML, Sklearn, ONNX runtime) by up to ~44x
~44x
0
500
60Est/Dep5 100Est/Dep4 100Est/Dep8 500Est/Dep8
Elapsed
time
(seconds)
Gradient Boost Models (Hospital 200M rows)
ONNX runtime Raven - CPU Raven - GPU
2500
3000
3500
End-to-end inference query time
Performance Evaluation: Raven in Spark with GPU
SELECT PREDICT(model, col1, …) FROM Hospital
Raven + GPU outperforms ONNX runtime by up to ~8x for complex models
~8x
1
10
100
1,000
10,000
100,000
DT-depth5 DT- depth8 LR-.001 GB/RF-20est DT-depth5 DT- depth8 LR-.001 GB-20est
hospital - 100M rows expedia - 100M rows
End-to-end
Time
(sec)
Log
Scale
End-to-end inference query time
MADlib SQL Server (DOP1) Raven (DOP1) SQL Server (DOP16) Raven (DOP16)
Performance Evaluation: Raven Plans in SQL Server
Potential gains with Raven in SQL Server are significantly large!
~230x
~100x
Best of Raven:
• Decision Trees (DT) and Logistic Regressions (LR): Model Projection Pushdown + ML-to-SQL
• Gradient Boost (GB): Model Projection Pushdown
Performance Evaluation: Raven in SQL Server with GPU
Potential gains with Raven and GPU acceleration are significantly large!
~100x
Batch size:
• CPU: Minimum query time obtained with optimal choice of batch size (50K/100K rows).
• GPU: 600K rows.
0
200
400
600
800
1000
1200
1400
depth3-
20est
depth5-
60est
depth4-
100est
depth8-
100est
depth8-
500est
End-to-end
Time
(secs)
Min. CPU-SKL GPU-HB
~2.6x
hospital – 100M rows, GB models
Demo
Conclusion: in-DBMS model inference
• Raven is the first step in a long journey of incorporating ML inference
as a foundational extension of relational algebra and an integral
part of SQL query optimizers and runtimes
• Novel Raven optimizer with cross optimizations and operator
transformations
Ø Up to 13x performance improvements on Spark
Ø Up to 330x performance improvements on SQL Server
• Integration of Raven within Spark and SQL Server
Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.
Backup
Current state of affairs: In-application model inference
Use case: hospital length-of-stay
“Find pregnant patients that are
expected to stay in the hospital more
than a week”
Security
• Data leaves the DB
• Model outside of the DB
Performance
• Data movement
• Use of Python for data operations
DBMS
Raven: In-DBMS model inference
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
Static
Analysis
MQ: inference query
Inference query: SQL + PREDICT (SQL Server
2017 syntax) to combine SQL operations with ML
inference
DBMS
DBMS
Raven
model
data
SQL
+
ML
Raven: In-DB model inference
DBMS
Raven
Security
• Data and models within the DB
• Treat models as data
User experience
• Leverage maturity of RDBMS
• Connectivity, tool integration
Can in-DBMS ML inference match (or exceed?)
the performance of state-of-the-art ML
frameworks?
Yes, by up to 230x!
Cross-optimizations in practice
l (name, model) AS
ay”,
ipeline import Pipeline
eprocessing import StandardScaler
ee import DecisionTreeClassifier
=
ion’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
nTreeClassifier())])”);
(Data Scientist)
oking model (Data Analyst)
varbinary(max) = (
FROM scoring_models
ame = ”duration_of_stay“ );
_info AS pi
ests AS be ON pi.id = be.id
l_tests AS pt ON be.id = pt.id
length_of_stay
DEL=@model, DATA=data AS d)
stay Pred float) AS p
t = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
y
Run
Cod
Cross-IR optimizations
and operator
transformations:
Ø Predicate-based
model pruning
Ø Model projection
pushdown
Ø Model splitting
Ø Model inlining
Ø NN translation
Ø Standard DB
optimizations
Ø Compiler
optimizations
Raven overview
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
patient_info blood_tests
Categorical
Encoding
FeatureExtractor
DecisionTreeClassifier
Rescaling
Concat
prenatal_tests
σ pregnant = 1
age
pregnant
gender
1 0
F M X
<35 >=35
…
bp … …
…
…
…
Unified IR for MQ
patient_info blood_tests
NeuralNet
prenatal_tests
Optimized plan for MQ
switch:
case (bp>140): 7
case (120<bp<140): 4
case (bp<120): 2
σage >35
σ pregnant = 1
π π π
σage <=35
U
σlength_of_stay >= 7
Static
Analysis
Cross
Optimization
2 4 7
… … … …
σlength_of_stay >= 7
σbp>140
SQL-inlined model
MQ: inference query
Runtime
Code gen
+
Key ideas:
1. Novel cross-optimizations between SQL and ML operations
2. Combine high-performance ML inference engines with SQL Server
Effect of cross optimizations
1
10
102
103
104
1K 10K 100K 1M
Inference
Time
(ms)
Log
Scale
Dataset Size
RF (scikit-learn)
RF-NN (CPU)
RF-NN (GPU)
24.5x
15x
5.3x
Execution modes
In-process
Deep integration of
ONNX Runtime in
SQL Server
Out-of-process
For queries/models not
supported by our static
analyzer
sp_execute_external_script
(Python, R, Java)
Containerized
For languages not
supported by out-
of-process execution
In-process execution
Native predict: execute the model in the same process as SQL Server
Rudimentary support since SQL Server 2017 (five hardcoded models)
Take advantage of state-of-the-art ML inference engines
Compiler optimizations, Code generation, Hardware acceleration
SQL Server + ONNX Runtime
Some challenges
Align schemata between DB and model
Transform data to/from tensors (avoid copying)
Cache inference sessions
Allow for different ML engines
INSERT INTO model (name, model) AS
(“duration_of_stay”,
“from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from …
model_pipeline =
Pipeline([(‘union’, FeatureUnion(…
(‘scaler’,StandardScaler()), …))
(‘clf’,DecisionTreeClassifier())])”);
M: model pipeline (Data Scientist)
Q: SQL query invoking model (Data Analyst)
DECLARE @model varbinary(max) = (
SELECT model FROM scoring_models
WHERE model_name = ”duration_of_stay“ );
WITH data AS(
SELECT *
FROM patient_info AS pi
JOIN blood_tests AS be ON pi.id = be.id
JOIN prenatal_tests AS pt ON be.id = pt.id
);
SELECT d.id, p.length_of_stay
FROM PREDICT(MODEL=@model, DATA=data AS d)
WITH(length_of_stay Pred float) AS p
WHERE d.pregnant = 1 AND p.length_of_stay > 7;
St
Ana
MQ: inference query
Current status
In-process predictions
Ø Implementation in SQL Server 2019
Ø Public preview in Azure SQL DB Edge
Ø Private preview in Azure SQL DW
Out-of-process predictions
Ø ONNX Runtime as an external
language
(ongoing)
Benefits of deep integration
1
10
100
1K
10k
1K 10K 100K 1M 10M 1K 10K 100K 1M 10M
Total
Inference
Time
(ms)
Log
Scale
Dataset Size
Random Forest MLP
ORT
Raven
Raven ext.

Weitere ähnliche Inhalte

Was ist angesagt?

Large Scale Lakehouse Implementation Using Structured Streaming
Large Scale Lakehouse Implementation Using Structured StreamingLarge Scale Lakehouse Implementation Using Structured Streaming
Large Scale Lakehouse Implementation Using Structured StreamingDatabricks
 
Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...
Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...
Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...Databricks
 
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3Iceberg: a fast table format for S3
Iceberg: a fast table format for S3DataWorks Summit
 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks FundamentalsDalibor Wijas
 
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...Databricks
 
ETL Using Informatica Power Center
ETL Using Informatica Power CenterETL Using Informatica Power Center
ETL Using Informatica Power CenterEdureka!
 
Koalas: Making an Easy Transition from Pandas to Apache Spark
Koalas: Making an Easy Transition from Pandas to Apache SparkKoalas: Making an Easy Transition from Pandas to Apache Spark
Koalas: Making an Easy Transition from Pandas to Apache SparkDatabricks
 
Hudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilitiesHudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilitiesNishith Agarwal
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptxAlex Ivy
 
Intro to Delta Lake
Intro to Delta LakeIntro to Delta Lake
Intro to Delta LakeDatabricks
 
PySpark dataframe
PySpark dataframePySpark dataframe
PySpark dataframeJaemun Jung
 
Learn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML LifecycleLearn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML LifecycleDatabricks
 
Technical Deck Delta Live Tables.pdf
Technical Deck Delta Live Tables.pdfTechnical Deck Delta Live Tables.pdf
Technical Deck Delta Live Tables.pdfIlham31574
 
What’s New with Databricks Machine Learning
What’s New with Databricks Machine LearningWhat’s New with Databricks Machine Learning
What’s New with Databricks Machine LearningDatabricks
 
Understanding Query Plans and Spark UIs
Understanding Query Plans and Spark UIsUnderstanding Query Plans and Spark UIs
Understanding Query Plans and Spark UIsDatabricks
 
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...Databricks
 
Building Data Quality pipelines with Apache Spark and Delta Lake
Building Data Quality pipelines with Apache Spark and Delta LakeBuilding Data Quality pipelines with Apache Spark and Delta Lake
Building Data Quality pipelines with Apache Spark and Delta LakeDatabricks
 
iceberg introduction.pptx
iceberg introduction.pptxiceberg introduction.pptx
iceberg introduction.pptxDori Waldman
 

Was ist angesagt? (20)

Large Scale Lakehouse Implementation Using Structured Streaming
Large Scale Lakehouse Implementation Using Structured StreamingLarge Scale Lakehouse Implementation Using Structured Streaming
Large Scale Lakehouse Implementation Using Structured Streaming
 
Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...
Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...
Presto: Fast SQL-on-Anything (including Delta Lake, Snowflake, Elasticsearch ...
 
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3Iceberg: a fast table format for S3
Iceberg: a fast table format for S3
 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks Fundamentals
 
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
 
ETL Using Informatica Power Center
ETL Using Informatica Power CenterETL Using Informatica Power Center
ETL Using Informatica Power Center
 
Koalas: Making an Easy Transition from Pandas to Apache Spark
Koalas: Making an Easy Transition from Pandas to Apache SparkKoalas: Making an Easy Transition from Pandas to Apache Spark
Koalas: Making an Easy Transition from Pandas to Apache Spark
 
Hudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilitiesHudi architecture, fundamentals and capabilities
Hudi architecture, fundamentals and capabilities
 
Databricks Platform.pptx
Databricks Platform.pptxDatabricks Platform.pptx
Databricks Platform.pptx
 
Intro to Delta Lake
Intro to Delta LakeIntro to Delta Lake
Intro to Delta Lake
 
PySpark dataframe
PySpark dataframePySpark dataframe
PySpark dataframe
 
Learn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML LifecycleLearn to Use Databricks for the Full ML Lifecycle
Learn to Use Databricks for the Full ML Lifecycle
 
Technical Deck Delta Live Tables.pdf
Technical Deck Delta Live Tables.pdfTechnical Deck Delta Live Tables.pdf
Technical Deck Delta Live Tables.pdf
 
What’s New with Databricks Machine Learning
What’s New with Databricks Machine LearningWhat’s New with Databricks Machine Learning
What’s New with Databricks Machine Learning
 
MapReduce
MapReduceMapReduce
MapReduce
 
Understanding Query Plans and Spark UIs
Understanding Query Plans and Spark UIsUnderstanding Query Plans and Spark UIs
Understanding Query Plans and Spark UIs
 
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...
 
Apache Hadoop 3
Apache Hadoop 3Apache Hadoop 3
Apache Hadoop 3
 
Building Data Quality pipelines with Apache Spark and Delta Lake
Building Data Quality pipelines with Apache Spark and Delta LakeBuilding Data Quality pipelines with Apache Spark and Delta Lake
Building Data Quality pipelines with Apache Spark and Delta Lake
 
iceberg introduction.pptx
iceberg introduction.pptxiceberg introduction.pptx
iceberg introduction.pptx
 

Ähnlich wie Raven: End-to-end Optimization of ML Prediction Queries

Translating data to predictive models
Translating data to predictive modelsTranslating data to predictive models
Translating data to predictive modelsChemAxon
 
Invited talk @Aberdeen, '07: Modelling and computing the quality of informati...
Invited talk @Aberdeen, '07: Modelling and computing the quality of informati...Invited talk @Aberdeen, '07: Modelling and computing the quality of informati...
Invited talk @Aberdeen, '07: Modelling and computing the quality of informati...Paolo Missier
 
Automation of building reliable models
Automation of building reliable modelsAutomation of building reliable models
Automation of building reliable modelsEszter Szabó
 
Certified Reasoning for Automated Verification
Certified Reasoning for Automated VerificationCertified Reasoning for Automated Verification
Certified Reasoning for Automated VerificationAsankhaya Sharma
 
Predictive Modeling Workshop
Predictive Modeling WorkshopPredictive Modeling Workshop
Predictive Modeling Workshopodsc
 
Translating data to model ICCS2022_pub.pdf
Translating data to model ICCS2022_pub.pdfTranslating data to model ICCS2022_pub.pdf
Translating data to model ICCS2022_pub.pdfwhitecomma
 
Data mining with caret package
Data mining with caret packageData mining with caret package
Data mining with caret packageVivian S. Zhang
 
Static analysis: Around Java in 60 minutes
Static analysis: Around Java in 60 minutesStatic analysis: Around Java in 60 minutes
Static analysis: Around Java in 60 minutesAndrey Karpov
 
Machinelearning Spark Hadoop User Group Munich Meetup 2016
Machinelearning Spark Hadoop User Group Munich Meetup 2016Machinelearning Spark Hadoop User Group Munich Meetup 2016
Machinelearning Spark Hadoop User Group Munich Meetup 2016Comsysto Reply GmbH
 
Lab 2: Classification and Regression Prediction Models, training and testing ...
Lab 2: Classification and Regression Prediction Models, training and testing ...Lab 2: Classification and Regression Prediction Models, training and testing ...
Lab 2: Classification and Regression Prediction Models, training and testing ...Yao Yao
 
EVERYTHING ABOUT STATIC CODE ANALYSIS FOR A JAVA PROGRAMMER
EVERYTHING ABOUT STATIC CODE ANALYSIS FOR A JAVA PROGRAMMEREVERYTHING ABOUT STATIC CODE ANALYSIS FOR A JAVA PROGRAMMER
EVERYTHING ABOUT STATIC CODE ANALYSIS FOR A JAVA PROGRAMMERAndrey Karpov
 
Nyc open-data-2015-andvanced-sklearn-expanded
Nyc open-data-2015-andvanced-sklearn-expandedNyc open-data-2015-andvanced-sklearn-expanded
Nyc open-data-2015-andvanced-sklearn-expandedVivian S. Zhang
 
Thesis presentation am lesas
Thesis presentation am lesasThesis presentation am lesas
Thesis presentation am lesasAnne-Marie Lesas
 
Advanced pg_stat_statements: Filtering, Regression Testing & more
Advanced pg_stat_statements: Filtering, Regression Testing & moreAdvanced pg_stat_statements: Filtering, Regression Testing & more
Advanced pg_stat_statements: Filtering, Regression Testing & moreLukas Fittl
 
Invited talk @Roma La Sapienza, April '07
Invited talk @Roma La Sapienza, April '07Invited talk @Roma La Sapienza, April '07
Invited talk @Roma La Sapienza, April '07Paolo Missier
 
Smart Data Conference: DL4J and DataVec
Smart Data Conference: DL4J and DataVecSmart Data Conference: DL4J and DataVec
Smart Data Conference: DL4J and DataVecJosh Patterson
 
Running Intelligent Applications inside a Database: Deep Learning with Python...
Running Intelligent Applications inside a Database: Deep Learning with Python...Running Intelligent Applications inside a Database: Deep Learning with Python...
Running Intelligent Applications inside a Database: Deep Learning with Python...Miguel González-Fierro
 
Mining attributes
Mining attributesMining attributes
Mining attributesSandra Alex
 

Ähnlich wie Raven: End-to-end Optimization of ML Prediction Queries (20)

Translating data to predictive models
Translating data to predictive modelsTranslating data to predictive models
Translating data to predictive models
 
Invited talk @Aberdeen, '07: Modelling and computing the quality of informati...
Invited talk @Aberdeen, '07: Modelling and computing the quality of informati...Invited talk @Aberdeen, '07: Modelling and computing the quality of informati...
Invited talk @Aberdeen, '07: Modelling and computing the quality of informati...
 
Presentation
PresentationPresentation
Presentation
 
Automation of building reliable models
Automation of building reliable modelsAutomation of building reliable models
Automation of building reliable models
 
Certified Reasoning for Automated Verification
Certified Reasoning for Automated VerificationCertified Reasoning for Automated Verification
Certified Reasoning for Automated Verification
 
Predictive Modeling Workshop
Predictive Modeling WorkshopPredictive Modeling Workshop
Predictive Modeling Workshop
 
Translating data to model ICCS2022_pub.pdf
Translating data to model ICCS2022_pub.pdfTranslating data to model ICCS2022_pub.pdf
Translating data to model ICCS2022_pub.pdf
 
Data mining with caret package
Data mining with caret packageData mining with caret package
Data mining with caret package
 
36x48_Trifold_FinalPoster
36x48_Trifold_FinalPoster36x48_Trifold_FinalPoster
36x48_Trifold_FinalPoster
 
Static analysis: Around Java in 60 minutes
Static analysis: Around Java in 60 minutesStatic analysis: Around Java in 60 minutes
Static analysis: Around Java in 60 minutes
 
Machinelearning Spark Hadoop User Group Munich Meetup 2016
Machinelearning Spark Hadoop User Group Munich Meetup 2016Machinelearning Spark Hadoop User Group Munich Meetup 2016
Machinelearning Spark Hadoop User Group Munich Meetup 2016
 
Lab 2: Classification and Regression Prediction Models, training and testing ...
Lab 2: Classification and Regression Prediction Models, training and testing ...Lab 2: Classification and Regression Prediction Models, training and testing ...
Lab 2: Classification and Regression Prediction Models, training and testing ...
 
EVERYTHING ABOUT STATIC CODE ANALYSIS FOR A JAVA PROGRAMMER
EVERYTHING ABOUT STATIC CODE ANALYSIS FOR A JAVA PROGRAMMEREVERYTHING ABOUT STATIC CODE ANALYSIS FOR A JAVA PROGRAMMER
EVERYTHING ABOUT STATIC CODE ANALYSIS FOR A JAVA PROGRAMMER
 
Nyc open-data-2015-andvanced-sklearn-expanded
Nyc open-data-2015-andvanced-sklearn-expandedNyc open-data-2015-andvanced-sklearn-expanded
Nyc open-data-2015-andvanced-sklearn-expanded
 
Thesis presentation am lesas
Thesis presentation am lesasThesis presentation am lesas
Thesis presentation am lesas
 
Advanced pg_stat_statements: Filtering, Regression Testing & more
Advanced pg_stat_statements: Filtering, Regression Testing & moreAdvanced pg_stat_statements: Filtering, Regression Testing & more
Advanced pg_stat_statements: Filtering, Regression Testing & more
 
Invited talk @Roma La Sapienza, April '07
Invited talk @Roma La Sapienza, April '07Invited talk @Roma La Sapienza, April '07
Invited talk @Roma La Sapienza, April '07
 
Smart Data Conference: DL4J and DataVec
Smart Data Conference: DL4J and DataVecSmart Data Conference: DL4J and DataVec
Smart Data Conference: DL4J and DataVec
 
Running Intelligent Applications inside a Database: Deep Learning with Python...
Running Intelligent Applications inside a Database: Deep Learning with Python...Running Intelligent Applications inside a Database: Deep Learning with Python...
Running Intelligent Applications inside a Database: Deep Learning with Python...
 
Mining attributes
Mining attributesMining attributes
Mining attributes
 

Mehr von Databricks

DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1Databricks
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Databricks
 
Data Lakehouse Symposium | Day 2
Data Lakehouse Symposium | Day 2Data Lakehouse Symposium | Day 2
Data Lakehouse Symposium | Day 2Databricks
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Databricks
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of HadoopDatabricks
 
Democratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized PlatformDemocratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized PlatformDatabricks
 
Learn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceLearn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceDatabricks
 
Why APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML MonitoringWhy APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML MonitoringDatabricks
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixThe Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixDatabricks
 
Stage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI IntegrationStage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI IntegrationDatabricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorchSimplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorchDatabricks
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on KubernetesScaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on KubernetesDatabricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesScaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesDatabricks
 
Sawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature AggregationsSawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature AggregationsDatabricks
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen SinkRedis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen SinkDatabricks
 
Re-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and SparkRe-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and SparkDatabricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache SparkProcessing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache SparkDatabricks
 
Massive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta LakeMassive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta LakeDatabricks
 
Machine Learning CI/CD for Email Attack Detection
Machine Learning CI/CD for Email Attack DetectionMachine Learning CI/CD for Email Attack Detection
Machine Learning CI/CD for Email Attack DetectionDatabricks
 

Mehr von Databricks (20)

DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptx
 
Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
 
Data Lakehouse Symposium | Day 2
Data Lakehouse Symposium | Day 2Data Lakehouse Symposium | Day 2
Data Lakehouse Symposium | Day 2
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
 
Democratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized PlatformDemocratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized Platform
 
Learn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceLearn to Use Databricks for Data Science
Learn to Use Databricks for Data Science
 
Why APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML MonitoringWhy APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML Monitoring
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixThe Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
 
Stage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI IntegrationStage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI Integration
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorchSimplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorch
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on KubernetesScaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on Kubernetes
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesScaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
 
Sawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature AggregationsSawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature Aggregations
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen SinkRedis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
 
Re-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and SparkRe-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and Spark
 
Processing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache SparkProcessing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache Spark
 
Massive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta LakeMassive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta Lake
 
Machine Learning CI/CD for Email Attack Detection
Machine Learning CI/CD for Email Attack DetectionMachine Learning CI/CD for Email Attack Detection
Machine Learning CI/CD for Email Attack Detection
 

Kürzlich hochgeladen

Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 

Kürzlich hochgeladen (17)

Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 

Raven: End-to-end Optimization of ML Prediction Queries

  • 1. Raven: End-to-end Optimization of ML Prediction Queries Konstantinos Karanasos, Kwanghyun Park Gray Systems Lab, Microsoft
  • 2. App logic offline online Model Inference Featurization Model Model optimization policies orchestratio n Data Catalogs Governance Model Tracking & Provenance Access Control Logs & Telemetry policies Decisions Live Data deployment other data featurizatio n Model Training Model Development / Training offline feat. Model Enterprise-grade ML lifecycle
  • 3. Data Scientist Analyst/Developer model training model scoring data exploration/ preparation data selection/ transformation model deployment Use Case: Length-of-stay in Hospital Model: “Predict length of stay of a patient in the hospital” Prediction query: “Find pregnant patients that are expected to stay in the hospital more than a week”
  • 4. Featurization Model Container REST Prediction Queries: Baseline Approach policies HTTP WebServer App logic ODBC DBMS Enterprise Features • Security: data and models outside of the DB • Extra infrastructure • High TCO • Lack tooling/best-practices Performance • Data movement • Latency • Throughput on batch-scoring
  • 5. Prediction Queries: In-Engine Evaluation policies HTTP WebServer App logic ODBC DBMS Enterprise Features • Security: Data and models within the DBMS • Reuse Existing infrastructure • Language/tools/best practices • Low TCO Performance ? • Up to 13x faster on Spark • Up to 330x faster on SQL Server
  • 6. Raven: An Optimizer for Prediction Queries in Azure Data + data models Unified IR INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Optimized IR INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Embed high-performance ML inference runtimes within our data engines Express data and ML operations in a common graph
  • 7. Constructing the IR Raven IR operators Relational algebra Linear algebra Other ML operators and data featurizers UDFs Static analysis of the prediction query Support for SQL+ML Adding support for Python INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 MQ: inference query
  • 8. ML Inference in Azure Data Engines SQL Server PREDICT statement in SQL Server Embedded ONNX Runtime in the engine Available in Azure SQL Edge and SQL DW (part of Azure Synapse Analytics) Spark Introduced a new PREDICT operator Similar syntax to SQL Server Support for different types of models INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; pa Un Static Analysis MQ: inference query
  • 9. INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Q: “Find pregnant patients expected to stay in the hospital more than a week” INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Raven: An Optimizer for Prediction Queries + + Runtime Code Gen
  • 10. Raven optimizations in practice (name, model) AS ”, eline import Pipeline rocessing import StandardScaler import DecisionTreeClassifier n’, FeatureUnion(… scaler’,StandardScaler()), …)) reeClassifier())])”); Data Scientist) ng model (Data Analyst) rbinary(max) = ( OM scoring_models e = ”duration_of_stay“ ); nfo AS pi ts AS be ON pi.id = be.id tests AS pt ON be.id = pt.id ngth_of_stay L=@model, DATA=data AS d) ay Pred float) AS p = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model Runti Code 1. Predicate-based model pruning 2. Model projection pushdown 3. Model splitting 4. Model-to-SQL translation 5. NN translation 6. Standard DB optimizations 7. Compiler optimizations
  • 11. 1. Avoid unnecessary computation Information passing between model and data 2. Pick the right runtime for each operation Translation between data and ML operations 3. Hardware acceleration Translation to tensor computations (Hummingbird) INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen Raven optimizations: Key Ideas
  • 12. 0 1,000 2,000 3,000 4,000 5,000 DT-depth5 DT-depth8 LR-.001 GB-20est DT-depth5 DT-depth8 LR-.001 GB-20est Hospital - 2 billion rows Expedia - 500 million rows Elapsed time (seconds) End-to-end inferene query time SparkML Sklearn ONNX runtime Raven Performance Evaluation: Raven in Spark (HDI) Best of Raven: • Decision Trees (DT) and Logistic Regressions (LR): Model Projection Pushdown + ML-to-SQL • Gradient Boost (GB): Model Projection Pushdown SELECT PREDICT(model, col1, …) FROM Hospital SELECT PREDICT(model, S.col1, …) FROM listings S, hotels R1, searches R2 WHERE S.prop_id = R1.prop_id AND S.srch_id = R2.srch_id Raven outperforms other ML runtimes (SparkML, Sklearn, ONNX runtime) by up to ~44x ~44x
  • 13. 0 500 60Est/Dep5 100Est/Dep4 100Est/Dep8 500Est/Dep8 Elapsed time (seconds) Gradient Boost Models (Hospital 200M rows) ONNX runtime Raven - CPU Raven - GPU 2500 3000 3500 End-to-end inference query time Performance Evaluation: Raven in Spark with GPU SELECT PREDICT(model, col1, …) FROM Hospital Raven + GPU outperforms ONNX runtime by up to ~8x for complex models ~8x
  • 14. 1 10 100 1,000 10,000 100,000 DT-depth5 DT- depth8 LR-.001 GB/RF-20est DT-depth5 DT- depth8 LR-.001 GB-20est hospital - 100M rows expedia - 100M rows End-to-end Time (sec) Log Scale End-to-end inference query time MADlib SQL Server (DOP1) Raven (DOP1) SQL Server (DOP16) Raven (DOP16) Performance Evaluation: Raven Plans in SQL Server Potential gains with Raven in SQL Server are significantly large! ~230x ~100x Best of Raven: • Decision Trees (DT) and Logistic Regressions (LR): Model Projection Pushdown + ML-to-SQL • Gradient Boost (GB): Model Projection Pushdown
  • 15. Performance Evaluation: Raven in SQL Server with GPU Potential gains with Raven and GPU acceleration are significantly large! ~100x Batch size: • CPU: Minimum query time obtained with optimal choice of batch size (50K/100K rows). • GPU: 600K rows. 0 200 400 600 800 1000 1200 1400 depth3- 20est depth5- 60est depth4- 100est depth8- 100est depth8- 500est End-to-end Time (secs) Min. CPU-SKL GPU-HB ~2.6x hospital – 100M rows, GB models
  • 16. Demo
  • 17. Conclusion: in-DBMS model inference • Raven is the first step in a long journey of incorporating ML inference as a foundational extension of relational algebra and an integral part of SQL query optimizers and runtimes • Novel Raven optimizer with cross optimizations and operator transformations Ø Up to 13x performance improvements on Spark Ø Up to 330x performance improvements on SQL Server • Integration of Raven within Spark and SQL Server
  • 18. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.
  • 20. Current state of affairs: In-application model inference Use case: hospital length-of-stay “Find pregnant patients that are expected to stay in the hospital more than a week” Security • Data leaves the DB • Model outside of the DB Performance • Data movement • Use of Python for data operations DBMS
  • 21. Raven: In-DBMS model inference INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; Static Analysis MQ: inference query Inference query: SQL + PREDICT (SQL Server 2017 syntax) to combine SQL operations with ML inference DBMS DBMS Raven model data SQL + ML
  • 22. Raven: In-DB model inference DBMS Raven Security • Data and models within the DB • Treat models as data User experience • Leverage maturity of RDBMS • Connectivity, tool integration Can in-DBMS ML inference match (or exceed?) the performance of state-of-the-art ML frameworks? Yes, by up to 230x!
  • 23. Cross-optimizations in practice l (name, model) AS ay”, ipeline import Pipeline eprocessing import StandardScaler ee import DecisionTreeClassifier = ion’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) nTreeClassifier())])”); (Data Scientist) oking model (Data Analyst) varbinary(max) = ( FROM scoring_models ame = ”duration_of_stay“ ); _info AS pi ests AS be ON pi.id = be.id l_tests AS pt ON be.id = pt.id length_of_stay DEL=@model, DATA=data AS d) stay Pred float) AS p t = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model y Run Cod Cross-IR optimizations and operator transformations: Ø Predicate-based model pruning Ø Model projection pushdown Ø Model splitting Ø Model inlining Ø NN translation Ø Standard DB optimizations Ø Compiler optimizations
  • 24. Raven overview INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; patient_info blood_tests Categorical Encoding FeatureExtractor DecisionTreeClassifier Rescaling Concat prenatal_tests σ pregnant = 1 age pregnant gender 1 0 F M X <35 >=35 … bp … … … … … Unified IR for MQ patient_info blood_tests NeuralNet prenatal_tests Optimized plan for MQ switch: case (bp>140): 7 case (120<bp<140): 4 case (bp<120): 2 σage >35 σ pregnant = 1 π π π σage <=35 U σlength_of_stay >= 7 Static Analysis Cross Optimization 2 4 7 … … … … σlength_of_stay >= 7 σbp>140 SQL-inlined model MQ: inference query Runtime Code gen + Key ideas: 1. Novel cross-optimizations between SQL and ML operations 2. Combine high-performance ML inference engines with SQL Server
  • 25. Effect of cross optimizations 1 10 102 103 104 1K 10K 100K 1M Inference Time (ms) Log Scale Dataset Size RF (scikit-learn) RF-NN (CPU) RF-NN (GPU) 24.5x 15x 5.3x
  • 26. Execution modes In-process Deep integration of ONNX Runtime in SQL Server Out-of-process For queries/models not supported by our static analyzer sp_execute_external_script (Python, R, Java) Containerized For languages not supported by out- of-process execution
  • 27. In-process execution Native predict: execute the model in the same process as SQL Server Rudimentary support since SQL Server 2017 (five hardcoded models) Take advantage of state-of-the-art ML inference engines Compiler optimizations, Code generation, Hardware acceleration SQL Server + ONNX Runtime Some challenges Align schemata between DB and model Transform data to/from tensors (avoid copying) Cache inference sessions Allow for different ML engines INSERT INTO model (name, model) AS (“duration_of_stay”, “from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier from … model_pipeline = Pipeline([(‘union’, FeatureUnion(… (‘scaler’,StandardScaler()), …)) (‘clf’,DecisionTreeClassifier())])”); M: model pipeline (Data Scientist) Q: SQL query invoking model (Data Analyst) DECLARE @model varbinary(max) = ( SELECT model FROM scoring_models WHERE model_name = ”duration_of_stay“ ); WITH data AS( SELECT * FROM patient_info AS pi JOIN blood_tests AS be ON pi.id = be.id JOIN prenatal_tests AS pt ON be.id = pt.id ); SELECT d.id, p.length_of_stay FROM PREDICT(MODEL=@model, DATA=data AS d) WITH(length_of_stay Pred float) AS p WHERE d.pregnant = 1 AND p.length_of_stay > 7; St Ana MQ: inference query
  • 28. Current status In-process predictions Ø Implementation in SQL Server 2019 Ø Public preview in Azure SQL DB Edge Ø Private preview in Azure SQL DW Out-of-process predictions Ø ONNX Runtime as an external language (ongoing)
  • 29. Benefits of deep integration 1 10 100 1K 10k 1K 10K 100K 1M 10M 1K 10K 100K 1M 10M Total Inference Time (ms) Log Scale Dataset Size Random Forest MLP ORT Raven Raven ext.