Telluride Workshop on Computational Materials Chemistry, Telluride, Colorado, USA, July 1, 2021.
https://research.chem.ucr.edu/groups/jiang/Telluride_Workshop.html
https://www.telluridescience.org/meetings/workshop-details?wid=901
https://www.telluridescience.org/meetings/workshop-details?wid=945
A machine-learning view on heterogeneous catalyst design and discovery
1. / 35
1
A machine-learning view on
heterogeneous catalyst design and discovery
Ichigaku Takigawa
ichigaku.takigawa@riken.jp
1 July 2021 @ Telluride, Colorado
Telluride Workshop on Computational Materials Chemistry
Advanced Intelligence Project
2. / 35
2
RIKEN Center for AI Project Inst. Chemical Reaction Design & Discovery
Hokkaido Univ
Two Interrelated Research Interests:
Hi, I am a ML researcher working for
ML for Stem Cell Biology ML for Chemistry
ML with discrete (combinatorial) structures ML for natural sciences
Edit-aware graph autocompletion (Hu+) Low-electron dose TEM image improvement (Katsuno+)
3. / 35
3
Todayâs talk
Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis)
https://en.wikipedia.org/wiki/Heterogeneous_catalysis
Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas PuriïŹcation, Methane Conversion, etc.
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
Adsorption
Diffusion
Dissociation
Recombination
Desorption
Our struggles for better ML practices with underspeciïŹed, sparse, biased
observational data (i.e. a collection of experimental facts from literature)
4. / 35
3
Todayâs talk
Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis)
https://en.wikipedia.org/wiki/Heterogeneous_catalysis
Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas PuriïŹcation, Methane Conversion, etc.
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
High Temperature, High Pressure
Adsorption
Diffusion
Dissociation
Recombination
Desorption
Our struggles for better ML practices with underspeciïŹed, sparse, biased
observational data (i.e. a collection of experimental facts from literature)
5. / 35
4
Todayâs talk
Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis)
Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas PuriïŹcation, Methane Conversion, etc.
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
High Temperature, High Pressure
Adsorption
Diffusion
Dissociation
Recombination
Desorption
God made the bulk;
the surface was invented by the devil
Devilishly complex too-many-factor process!!
ââ Wolfgang Pauli
Our struggles for better ML practices with underspeciïŹed, sparse, biased
observational data (i.e. a collection of experimental facts from literature)
6. / 35
5
simulation
input output
ML: A new way for (lazy) programming
computer program
input output
computer program
ML
full speciïŹcation in
every detail required
give up explicit model
instead, grab a tunable
model, and show it many
input-output instances
inductive (empiricism)
airhead with a god-like
learning capability
Random Forest Neural Networks SVR Kernel Ridge
All about ïŹtting a very-ïŹexible function to ïŹnite points in high-dimensional space.
deductive (rationalism)
7. / 35
6
ML: A new way for (lazy) programming
ResNet50: 26 million params
ResNet101: 45 million params
EïŹcientNet-B7: 66 million params
VGG19: 144 million params
12-layer, 12-heads BERT: 110 million params
24-layer, 16-heads BERT: 336 million params
GPT-2 XL: 1558 million params
GPT-3: 175 billion params
simulation
input output
computer program
input output
computer program
ML
full speciïŹcation in
every detail required
give up explicit model
instead, grab a tunable
model, and show it many
input-output instances
inductive (empiricism)
airhead with a god-like
learning capability
deductive (rationalism)
All about ïŹtting a very-ïŹexible function to ïŹnite points in high-dimensional space.
8. / 35
6
ML: A new way for (lazy) programming
ResNet50: 26 million params
ResNet101: 45 million params
EïŹcientNet-B7: 66 million params
VGG19: 144 million params
12-layer, 12-heads BERT: 110 million params
24-layer, 16-heads BERT: 336 million params
GPT-2 XL: 1558 million params
GPT-3: 175 billion params
Modern ML: Can we imagine what would happen if we try to ïŹt a function having 175
billion parameters to 100 million data points in 10 thousand dimension??
simulation
input output
computer program
input output
computer program
ML
full speciïŹcation in
every detail required
give up explicit model
instead, grab a tunable
model, and show it many
input-output instances
inductive (empiricism)
airhead with a god-like
learning capability
deductive (rationalism)
All about ïŹtting a very-ïŹexible function to ïŹnite points in high-dimensional space.
9. / 35
7
Rashomon Effect: multiplicity of good models
ML models are too ïŹexible to overrepresent given ïŹnite instances, and many different
shapes of functions exist for representing the same ïŹnite data. (even if itâs huge)
The Rashomon Effect
In many practical cases, we have many equally-accurate but different ML models.
(the choice of ML methods or the design of NN architectures doesnât really matter)
Note: The Rashomon Effect in ML is attributed to Leo Breimanâs very inïŹuential âThe Two Culturesâ paper published in
2001, but obviously âRashomonâ itself originates from a classic Japanese movie in 1950 by Kurosawa, where four
witnesses to a murder describe entirely different contradictory perspectives, but all of them sound true.
We often see this in ML competitions. Top ranking solutions are very competitive in
performance (equally accurate in practice) but can be very different approaches.
5-CV RMSE: 0.12016 5-CV RMSE: 0.13209 5-CV RMSE: 0.11976 5-CV RMSE: 0.12432 5-CV RMSE: 0.20899 5-CV RMSE: 0.17446
Neural Network Random Forest ExtraTrees GBDT Gaussian Process Kernel Ridge
10. / 35
8
We see differences in underspecified cases
right data scarce/underspeciïŹed + outliers
Neural Networks (ReLU)
Random Forest
Extra Trees (bootstrap)
Neural Networks (ReLU)
Random Forest
Extra Trees (bootstrap)
Neural Networks (Tanh)
Linear Regression
Kernel Ridge (RBF) Kernel Ridge (Laplacian)
Extra Trees (no bootstrap)
Gradient Boosting
11. / 35
8
We see differences in underspecified cases
right data scarce/underspeciïŹed + outliers
Neural Networks (ReLU)
Random Forest
Extra Trees (bootstrap)
Neural Networks (ReLU)
Random Forest
Extra Trees (bootstrap)
Neural Networks (Tanh)
Linear Regression
Kernel Ridge (RBF) Kernel Ridge (Laplacian)
Extra Trees (no bootstrap)
Gradient Boosting
some apparently underïŹtted?
12. / 35
8
We see differences in underspecified cases
right data scarce/underspeciïŹed + outliers
Neural Networks (ReLU)
Random Forest
Extra Trees (bootstrap)
Neural Networks (ReLU)
Random Forest
Extra Trees (bootstrap)
Neural Networks (Tanh)
Linear Regression
Kernel Ridge (RBF) Kernel Ridge (Laplacian)
Extra Trees (no bootstrap)
Gradient Boosting
some apparently underïŹtted?
But we often still see the Rashomon
(i.e. similar CV performances)
and these can predict very differently for
further test cases. (in particular, out-of-
distribution cases)
13. / 35
9
Designing relevant âinductive biasesâ
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
Prediction
Input
variables
ClassiïŹer or
Regressor
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Input representation design,
selection, and its engineering
(descriptors + feature engineering)
Function
model
vs âKitchen-sinkâ models
(feeding every possible descriptor)
Simple model is enough whenever we can have determining input variables necessary
and suïŹcient to fully deïŹne the desired output.
Every requirement should be
explicitly encoded into model,
otherwise we need to provide it
through examples.
(symmetry, invariance, etc)
14. / 35
10
Designing relevant âinductive biasesâ
Prediction
Input
variables
Function
model
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
ClassiïŹer or
Regressor
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
Architecture design when we go for representation learning
âKitchen-sinkâ
(raw) inputs
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
15. / 35
10
Designing relevant âinductive biasesâ
Prediction
Input
variables
Function
model
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
ClassiïŹer or
Regressor
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
Architecture design when we go for representation learning
âKitchen-sinkâ
(raw) inputs
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
16. / 35
10
Designing relevant âinductive biasesâ
Prediction
Input
variables
Function
model
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
ClassiïŹer or
Regressor
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
Architecture design when we go for representation learning
âKitchen-sinkâ
(raw) inputs
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
17. / 35
10
Designing relevant âinductive biasesâ
Prediction
Input
variables
Function
model
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
ClassiïŹer or
Regressor
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
Architecture design when we go for representation learning
âKitchen-sinkâ
(raw) inputs
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
18. / 35
10
Designing relevant âinductive biasesâ
Prediction
Input
variables
Function
model
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
ClassiïŹer or
Regressor
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
Architecture design when we go for representation learning
âKitchen-sinkâ
(raw) inputs
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
19. / 35
10
Designing relevant âinductive biasesâ
Prediction
Input
variables
Function
model
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
ClassiïŹer or
Regressor
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
Architecture design when we go for representation learning
âKitchen-sinkâ
(raw) inputs
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
20. / 35
10
Designing relevant âinductive biasesâ
Prediction
Input
variables
Function
model
<latexit sha1_base64="QFtMwnKe2I12XGZu0bNJbdnDaaE=">AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhAVIwV0caShzwSJGR3HXHDvrK7EJH4Aya2UlhpYmH8AD/Axh+w4BOMJSY2Fl6WTYwS8W5m58yZe+6cmSuZqmI7jHV9wtj4xOSUfzowMzs3HwwtLBZso2HJPC8bqmGVJNHmqqLzvKM4Ki+ZFhc1SeVFqb7X3y82uWUrhn7gtExe0cSarhwrsugQlTutJqqhCIsxN8LDIO6BCLxIG6FHHOIIBmQ0oIFDh0NYhQibvjLiYDCJq6BNnEVIcfc5zhEgbYOyOGWIxNbpX6NV2WN1Wvdr2q5aplNUGhYpw4iyF3bPeuyZPbBX9vlnrbZbo++lRbM00HKzGrxYzn38q9JodnDyrRrp2cExtl2vCnk3XaZ/C3mgb551ermdbLS9xm7ZG/m/YV32RDfQm+/yXYZnr0f4kcgLvRg1KP67HcOgkIjFt2KJzEYkteu1yo8VrGKd+pFECvtII0/1a7jEFTqCX4gJm0JykCr4PM0SfoSQ+gJWLpCb</latexit>
x2
<latexit sha1_base64="lFhRrRrVTrFR31ebbMgRp5myJpc=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqcljfKgRCLMCeCwyDqghDcSOqBRxzhGDpkNFAHhwabsAoRFn1FRMFgEFdCmziTkOLsc5zDR9oGZXHKEImt0b9Kq6LLarTu17QctUynqDRMUgYRZi/snvXYM3tgr+zzz1ptp0bfS4tmaaDlRtl/sZz5+FdVp9nGybdqpGcbFew4XhXybjhM/xbyQN886/Qyu+lwe43dsjfyf8O67IluoDXf5bsUT1+P8CORF3oxalD0dzuGQS4WiW5HYqnNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFWE6QnA==</latexit>
x3
<latexit sha1_base64="0IPXcU0UIDvzZlYURjV2A/THv9U=">AAACiXichVG7SgNBFD2ur/hM1EawEYNiFWZFNKQKprGMj0TBBNndTHR0X+xOFmLwB6zsRK0ULMQP8ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7um8CVjz11Kd09vX39sYHBoeGQ0nhgbL/pOzTN4wXBMx9vWNZ+bwuYFKaTJt12Pa5Zu8i39MNfc3wq45wvH3pR1l5ctbc8WVWFokqhiKag40t9NJFmKhTHdDtQIJBFF3knco4QKHBiowQKHDUnYhAafvh2oYHCJK6NBnEdIhPscxxgkbY2yOGVoxB7Sf49WOxFr07pZ0w/VBp1i0vBIOY1Z9sRu2Rt7ZHfshX3+WasR1mh6qdOst7Tc3Y2fTG58/KuyaJbY/1Z19CxRRTr0Ksi7GzLNWxgtfXB09raRWZ9tzLFr9kr+r9gze6Ab2MG7cbPG1y87+NHJC70YNUj93Y52UFxIqUuphbXFZHYlalUMU5jBPPVjGVmsIo8C1T/AKc5xoQwpqpJWMq1UpSvSTOBHKLkvAi+SPA==</latexit>
.
.
.
Latent
variables
Learnable variable
transformation
Representation learning
ClassiïŹer or
Regressor
Linear
<latexit sha1_base64="Ill3Als4zZd947f5Xm9sW99d0QA=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTomJcEd245CGPBAlp64gNpW3aQkTiD5i4lYUrTVwYP8APcOMPuOATjEtM3LjwUpoYJeJtpnPmzD13zsyVTU21Hca6PmFsfGJyyj8dmJmdmw+GFhbzttGwFJ5TDM2wirJkc03Vec5RHY0XTYtLdVnjBbm2198vNLllq4Z+4LRMXq5LVV09VhXJISp7WhEroQiLMTfCw0D0QARepIzQIw5xBAMKGqiDQ4dDWIMEm74SRDCYxJXRJs4ipLr7HOcIkLZBWZwyJGJr9K/SquSxOq37NW1XrdApGg2LlGFE2Qu7Zz32zB7YK/v8s1bbrdH30qJZHmi5WQleLGc//lXVaXZw8q0a6dnBMbZdryp5N12mfwtloG+edXrZnUy0vcZu2Rv5v2Fd9kQ30Jvvyl2aZ65H+JHJC70YNUj83Y5hkI/HxK1YPL0RSe56rfJjBatYp34kkMQ+UshR/SoucYWO4BdiwqaQGKQKPk+zhB8hJL8AVA6Qmg==</latexit>
x1
Architecture design when we go for representation learning
âKitchen-sinkâ
(raw) inputs
Again, simple model is enough
when we have good features.
Use heuristic assumptions, domain knowledge to constrain/regularize the model space.
21. / 35
11
A toy example: approximate adders
Try to teach ML âarithmetic additionâ only by examples.
We can also add both 5+6 and 6+5 to tell ML
by examples that addition is commutative.
adder
<latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit>
x1
<latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit>
x2
<latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit>
x1 + x2
<latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit>
1 + 3 = 4
<latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit>
2 + 5 = 7
<latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit>
5 + 9 = 14
<latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit>
3 + 10 = 13
<latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit>
5 + 6 = 11
<latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit>
1 + 1 =?
<latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit>
1 + ( 1) =?
<latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit>
12892 + 9837 =?
<latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit>
2
<latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit>
0
<latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit>
22729
train
test
(the case where a clear answer and underlying logic exist)
22. / 35
12
A toy example: approximate adders
RF says 1 + 1 = 5.15 and 1 - 1 = 5.15 and 12892 + 9837 = 12.75
MLP better? But anyway itâs totally wrong.
2-10-5-1 Feed Forward NN (MLP, Multi-Layer Perceptron)
adder
<latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit>
x1
<latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit>
x2
<latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit>
x1 + x2
<latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit>
1 + 3 = 4
<latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit>
2 + 5 = 7
<latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit>
5 + 9 = 14
<latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit>
3 + 10 = 13
<latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit>
5 + 6 = 11
<latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit>
1 + 1 =?
<latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit>
1 + ( 1) =?
<latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit>
12892 + 9837 =?
<latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit>
2
<latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit>
0
<latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit>
22729
train
test
(the case where a clear answer and inductive logic exist)
Try to teach ML âarithmetic additionâ only by examples.
23. / 35
12
A toy example: approximate adders
RF says 1 + 1 = 5.15 and 1 - 1 = 5.15 and 12892 + 9837 = 12.75
MLP better? But anyway itâs totally wrong.
2-10-5-1 Feed Forward NN (MLP, Multi-Layer Perceptron)
adder
<latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit>
x1
<latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit>
x2
<latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit>
x1 + x2
<latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit>
1 + 3 = 4
<latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit>
2 + 5 = 7
<latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit>
5 + 9 = 14
<latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit>
3 + 10 = 13
<latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit>
5 + 6 = 11
<latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit>
1 + 1 =?
<latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit>
1 + ( 1) =?
<latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit>
12892 + 9837 =?
<latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit>
2
<latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit>
0
<latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit>
22729
train
test
(the case where a clear answer and inductive logic exist)
Try to teach ML âarithmetic additionâ only by examples.
25. / 35
13
A toy example: approximate adders
Linear regression rocks! đ
LR says 1 + 1 = 2 and 1 - 1 = 0 and 12892 + 9837 = 22729
All are because of âinductive biasâ intrinsically encoded in the model.
LR with two input variables is just
ïŹtting a plane
<latexit sha1_base64="iFKbOGXhRe3/O37wsq+JtGPu+ec=">AAAClnichVHLSsNAFD2Nr1pfrW4EN8FSEYQyUVFxIaKILvuwraBSkjitoWkSkrRaiz/gD7gQFwoq4gf4AW78ARf9BHFZwY0Lb9OAqKg3JPfMmXtuzsxVLF1zXMYaAaGjs6u7J9gb6usfGBwKR4azjlmxVZ5RTd20txTZ4bpm8IyruTrfsmwulxWd55TSams/V+W2o5nGpluz+G5ZLhpaQVNll6h8OCKLh3lJnBIVytOU1Xw4yuLMC/EnkHwQhR8JM3yPHezBhIoKyuAw4BLWIcOhZxsSGCzidlEnziakefscxwiRtkJVnCpkYkv0LdJq22cNWrd6Op5apb/o9NqkFBFjT+yWNdkju2PP7P3XXnWvR8tLjbLS1nIrP3Qymn77V1Wm7GL/U/WnZxcFLHheNfJueUzrFGpbXz06baYXU7H6BLtkL+T/gjXYA53AqL6qV0meOvvDj0Je6MZoQNL3cfwE2em4NBefSc5Gl1f8UQUxhnFM0jzmsYwNJJCh/gc4xzVuhFFhSVgT1tulQsDXjOBLCIkPAxKUnQ==</latexit>
ax1 + bx2 + c
Any three instances are enough to have
<latexit sha1_base64="57verNgszw6m8+AhOWc3Hu105bk=">AAACp3ichVFNS9xQFD2mftX6MW03BTfBwaIow42WtgiC6MadOjrjgMrwEp/jw0wSkjdT7eBe/AMuXCm4KAW3unfjH+jCn1C6VHDjojeZgKioN+S+88675+a8XDtwVaSJrlqMN61t7R2db7vedff09mXefyhGfi10ZMHxXT8s2SKSrvJkQSvtylIQSlG1Xblsb83E58t1GUbK95b0TiDXqqLiqQ3lCM1UOTMgJq1R046TM0mreVXZ1CIM/R/mdtkyRziPlTNZylES5lNgpSCLNOb9zBlWsQ4fDmqoQsKDZuxCIOJnBRYIAXNraDAXMlLJucQuulhb4yrJFYLZLc4V3q2krMf7uGeUqB3+istvyEoTg/SHftE1XdJv+kt3z/ZqJD1iLzu82k2tDMp9+58Wb19VVXnV2LxXvehZYwPfE6+KvQcJE9/CaerrPw+uFyfyg43PdEz/2P8RXdEF38Cr3zgnCzJ/+IIfm73wH+MBWY/H8RQUx3LW19z4wpfs1HQ6qk70YwBDPI9vmMIs5lHg/ns4xRnOjWFjzigapWap0ZJqPuJBGOI/dKSbIg==</latexit>
a = 1, b = 1, c = 0 ) x1 + x2
MLP has partial âlinearityâ inside and thatâs why MLP is better than RF is.
RF is âpiecewise constantâ and only returns values between sample min and max.
to points in 3D
adder
<latexit sha1_base64="tiacEhQgmTkomNeV5LHmrY6hmbk=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCJTXxhXRDcuecgjQULaOmJDaZu2EJH4AyZuZeFKExfGD/AD3PgDLvgE4xITNy68lCZGiXib6Zw5c8+dM3NlU1Nth7GOTxgZHRuf8E8GpqZnZoOhufmcbdQthWcVQzOsgizZXFN1nnVUR+MF0+JSTdZ4Xq7u9fbzDW7ZqqEfOE2Tl2pSRVePVUVyiMqclsVyKMJizI3wIBA9EIEXSSP0iEMcwYCCOmrg0OEQ1iDBpq8IEQwmcSW0iLMIqe4+xzkCpK1TFqcMidgq/Su0KnqsTuteTdtVK3SKRsMiZRhR9sLuWZc9swf2yj7/rNVya/S8NGmW+1puloMXi5mPf1U1mh2cfKuGenZwjG3Xq0reTZfp3ULp6xtn7W5mJx1trbBb9kb+b1iHPdEN9Ma7cpfi6eshfmTyQi9GDRJ/t2MQ5NZi4lZsPbURSex6rfJjCctYpX7EkcA+kshS/QoucYW24BdiwqYQ76cKPk+zgB8hJL4AVGCQmw==</latexit>
x1
<latexit sha1_base64="4Sn0JXQ8Nli9zYaRMiYfd4a9JHg=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBuCIDPjCuiG5c8pBHgoS0dcCG0jZtISLxB0zcysKVJi6MH+AHuPEHXPAJxiUmblx4KU2MEvE20zlz5p47Z+ZKhqpYNmNdjzA2PjE55Z32zczOzfsDC4s5S2+YMs/KuqqbBUm0uKpoPGsrtsoLhsnFuqTyvFTb7+/nm9y0FF07tFsGL9XFqqZUFFm0icqclmPlQIhFmBPBYRB1QQhuJPXAI45wDB0yGqiDQ4NNWIUIi74iomAwiCuhTZxJSHH2Oc7hI22DsjhliMTW6F+lVdFlNVr3a1qOWqZTVBomKYMIsxd2z3rsmT2wV/b5Z622U6PvpUWzNNByo+y/WM58/Kuq02zj5Fs10rONCnYcrwp5Nxymfwt5oG+edXqZ3XS4vcZu2Rv5v2Fd9kQ30Jrv8l2Kp69H+JHIC70YNSj6ux3DIBeLRLcjG6nNUGLPbZUXK1jFOvUjjgQOkESW6ldxiSt0BK8QEbaE+CBV8LiaJfwIIfEFVoCQnA==</latexit>
x2
<latexit sha1_base64="SKy45aVsIvNNIc9wr1Q6tiMY1uI=">AAACinichVHLSsNAFL2Nr1qrrboR3BRLRRDKpIrPTVEXLvuwD6glJHFah+ZFkpbW4g+4cyXYlYIL8QP8ADf+gIt+gris4MaFN2lAtFhvmMyZM/fcOTNXMhRm2YR0fdzI6Nj4hH8yMBWcngmFZ+fyll43ZZqTdUU3i5JoUYVpNGczW6FFw6SiKim0INX2nf1Cg5oW07Uju2XQsipWNVZhsmgjVWgK/GpTSAjhKIkTNyKDgPdAFLxI6eFHOIYT0EGGOqhAQQMbsQIiWPiVgAcCBnJlaCNnImLuPoVzCKC2jlkUM0Rka/iv4qrksRqunZqWq5bxFAWHicoIxMgLuSc98kweyCv5/LNW263heGnhLPW11BBCFwvZj39VKs42nH6rhnq2oQJbrleG3g2XcW4h9/WNs6tedicTay+TW/KG/m9IlzzhDbTGu3yXppnOED8SesEXwwbxv9sxCPKJOL8RX0uvR5N7Xqv8sAhLsIL92IQkHEIKcm79S7iGDhfkEtw2t9tP5XyeZh5+BHfwBX8Zkfc=</latexit>
x1 + x2
<latexit sha1_base64="1coS+Mt9eL/r4jEoYDkZ5NYfoeE=">AAACiHichVHLSsNAFD3Gd3206kZwUywVQSg3KlYFQerGZVutCiqSxGkdTJOQpIVa/AE3LlVcKbgQP8APcOMPuPATxKWCGxfepgFRUW+YzJkz99w5M1d3TOn5RI8tSmtbe0dnV3ekp7evPxobGFzz7IpriIJhm7a7oWueMKUlCr70TbHhuEIr66ZY1/eXGvvrVeF60rZW/ZojtstayZJFaWg+UwV1YmpheieWoBQFEf8J1BAkEEbWjt1iC7uwYaCCMgQs+IxNaPD424QKgsPcNurMuYxksC9wiAhrK5wlOENjdp//JV5thqzF60ZNL1AbfIrJw2VlHEl6oGt6oXu6oSd6/7VWPajR8FLjWW9qhbMTPRpeeftXVebZx96n6k/PPoqYDbxK9u4ETOMWRlNfPTh5WZnPJ+tjdEnP7P+CHumOb2BVX42rnMif/+FHZy/8Ytwg9Xs7foK1yZQ6k5rKTScWM2GrujCCUYxzP9JYxDKyKHB9iWOc4kyJKKSklblmqtISaobwJZTMB48DkKc=</latexit>
1 + 3 = 4
<latexit sha1_base64="Jl1LVcl+mtrFJMn2AKjNmRINEM4=">AAACiHichVHLSsNAFD3GV62vqhvBjVgUQSi39VEVhFI3LtVaW1CRJE7rYJqEJC3U4g+4caniSsGF+AF+gBt/wIWfIC4ruHHhbRoQFfWGyZw5c8+dM3M125CuR/TUorS2tXd0hrrC3T29ff2RgcFN1yo7usjqlmE5eU11hSFNkfWkZ4i87Qi1pBkipx0sN/ZzFeG40jI3vKotdkpq0ZQFqaseU9nE1OxScjcSpRj5MfoTxAMQRRCrVuQO29iDBR1llCBgwmNsQIXL3xbiINjM7aDGnMNI+vsCRwiztsxZgjNUZg/4X+TVVsCavG7UdH21zqcYPBxWjmKcHumG6vRAt/RM77/Wqvk1Gl6qPGtNrbB3+4+HM2//qko8e9j/VP3p2UMB875Xyd5tn2ncQm/qK4en9czi+nhtgq7ohf1f0hPd8w3Myqt+vSbWL/7wo7EXfjFuUPx7O36CzUQsPhebXpuJptJBq0IYwRgmuR9JpLCCVWS5vsQJznCuhBVSkspCM1VpCTRD+BJK+gOby5Ct</latexit>
2 + 5 = 7
<latexit sha1_base64="NRtmyWkgZ5nlm1ETxVkZxPMO8Rc=">AAACiXichVHNLgNRGD0df6V+io3ERjREImnuUJREIrqxpNWSIM3MuGV0/jJz24TGC1jZCVYkFuIBPICNF7DwCGJZiY2Fb6aTCKK+yZ177rnf+e6591MdQ/cEY88RqaW1rb0j2tkV6+7p7Yv3DxQ8u+JqPK/Zhu1uqorHDd3ieaELg286LldM1eAbajnj729UuevptrUuDh2+Yyp7ll7SNUUQVZiZnF+UU8V4giVZECO/gRyCBMJYteP32MYubGiowASHBUHYgAKPvi3IYHCI20GNOJeQHuxzHKOLtBXK4pShEFum/x6ttkLWorVf0wvUGp1i0HBJOYIx9sRuWZ09sjv2wj7+rFULavheDmlWG1ruFPtOhnLv/6pMmgX2v1RNPQuUkA686uTdCRj/FlpDXz06q+cWsmO1cXbNXsn/FXtmD3QDq/qm3azx7GUTPyp5oRejBsk/2/EbFKaS8mxyei2VWFoOWxXFMEYxQf2YwxJWsIo81T/AKc5xIcUkWUpLC41UKRJqBvEtpMwnMx2Q7A==</latexit>
5 + 9 = 14
<latexit sha1_base64="wFMJb33bMt0hW8th8BMIqvUdNU8=">AAACinichVFLSwJRFD5OLzNLq03QRhIjCOSMRk8CqRYtfeQDTGRmutrgvJgZBZP+QLtWQa4KWkQ/oB/Qpj/Qwp8QLQ3atOg4DkRJdoY797vfPd+5371HNBTZshE7Hm5kdGx8wjvpm/JPzwSCs3M5S6+bEstKuqKbBVGwmCJrLGvLtsIKhskEVVRYXqzt9/bzDWZasq4d2U2DlVShqskVWRJsovLxVR53+Xg5GMYoOhEaBLwLwuBGUg8+wjGcgA4S1EEFBhrYhBUQwKKvCDwgGMSVoEWcSUh29hmcg4+0dcpilCEQW6N/lVZFl9Vo3atpOWqJTlFomKQMQQRf8B67+IwP+Iqff9ZqOTV6Xpo0i30tM8qBi4XMx78qlWYbTr9VQz3bUIFNx6tM3g2H6d1C6usbZ1fdzHY60lrGW3wj/zfYwSe6gdZ4l+5SLN0e4kckL/Ri1CD+dzsGQS4W5dej8dRaOLHntsoLi7AEK9SPDUjAISQh69S/hGtoc34uxm1xO/1UzuNq5uFHcAdfqESRGw==</latexit>
3 + 10 = 13
<latexit sha1_base64="FoI7hqDrtQtFjlgLPgriDwtw1iU=">AAACiXichVHLSsNAFD2Nr1pfVTeCm2JRBKFMfNQiCKIbl33YVlApSRx1NE1CMi1o8QdcuRN1peBC/AA/wI0/4MJPEJcV3LjwNg2IFvWGyZw5c8+dM3N1xxSeZOw5pLS1d3R2hbsjPb19/QPRwaGCZ1dcg+cN27TddV3zuCksnpdCmnzdcblW1k1e1A9WGvvFKnc9YVtr8tDhW2Vt1xI7wtAkUYW5qeSiqpaicZZgfsRagRqAOIJI29F7bGIbNgxUUAaHBUnYhAaPvg2oYHCI20KNOJeQ8Pc5jhEhbYWyOGVoxB7Qf5dWGwFr0bpR0/PVBp1i0nBJGcM4e2K3rM4e2R17YR+/1qr5NRpeDmnWm1rulAZORnLv/6rKNEvsfan+9Cyxg5TvVZB3x2catzCa+urRWT23kB2vTbBr9kr+r9gze6AbWNU34ybDs5d/+NHJC70YNUj92Y5WUJhOqMnETGY2vrQctCqMUYxhkvoxjyWsIo081d/HKc5xofQoqpJSFpqpSijQDONbKCufJlSQ5g==</latexit>
5 + 6 = 11
<latexit sha1_base64="n3Yy/9dJfdSB5yYDzUvImVPJv/g=">AAACiHichVHLSsNAFD3Gd3206kZwIxZFEMqNii8QRTcu1VotVClJnNahaRKStKDFH3DjUsWVggvxA/wAN/6Ai36CuFRw48LbNCAq1hsmc+bMPXfOzNUdU3o+UbVJaW5pbWvv6Ix0dff0RmN9/dueXXINkTJs03bTuuYJU1oi5UvfFGnHFVpRN8WOXlit7e+UhetJ29ryDx2xV9TylsxJQ/OZSqkT6uJSNhanBAUx/BuoIYgjjHU7do9d7MOGgRKKELDgMzahweMvAxUEh7k9VJhzGclgX+AYEdaWOEtwhsZsgf95XmVC1uJ1raYXqA0+xeThsnIYo/REt/RKj3RHz/TxZ61KUKPm5ZBnva4VTjZ6Mph8/1dV5NnHwZeqoWcfOcwFXiV7dwKmdgujri8fnb0mFzZHK2N0TS/s/4qq9MA3sMpvxs2G2Lxs4EdnL/xi3CD1Zzt+g+3JhDqTmNqYji+vhK3qwBBGMM79mMUy1rCOFNeXOMU5LpSIQsqsMl9PVZpCzQC+hbLyCaIfkLA=</latexit>
1 + 1 =?
<latexit sha1_base64="hRdVP+e2y4bjP1zOby3ZyPytH4w=">AAACi3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKRRhqa+gaatraxwsoG+gZgIECJsMQylBmgIKAfIHtDDEMKQz5DMkMpQy5DKkMeQwlQHYOQyJDMRBGMxgyGDAUAMViGaqBYkVAViZYPpWhloELqLcUqCoVqCIRKJoNJNOBvGioaB6QDzKzGKw7GWhLDhAXAXUqMKgaXDVYafDZ4ITBaoOXBn9wmlUNNgPklkognQTRm1oQz98lEfydoK5cIF3CkIHQhdfNJQxpDBZgt2YC3V4AFgH5Ihmiv6xq+udgqyDVajWDRQavge5faHDT4DDQB3llX5KXBqYGzcbjniSgW4AhBowgQ/TowGSEGekZmukZB5ooOzhBo4qDQZpBiUEDGB/mDA4MHgwBDKHgeJjEMJthDhMvkzGTFZMNRCkTI1SPMAMKYHIFACGukUw=</latexit>
1 + ( 1) =?
<latexit sha1_base64="iSckyhI7u8UoVI74y32YMTkgc7c=">AAACj3ichVHLSsNAFD3Gd3201Y3gRiwVQSiTVmwrqEU3umutVUFFkjitoWkSkmlBiz/gB+jChQ9wIX6AH+DGH3DhJ4jLCm5ceJsGREW9YTJnztxz58xc1TZ0VzD21Ca1d3R2dff0Bvr6BwaDofDQumtVHY0XNMuwnE1Vcbmhm7wgdGHwTdvhSkU1+IZaXmrub9S44+qWuSYObL5TUUqmXtQ1RRC1LcdT6fhUOpVIzi3shiIsxrwY+wlkH0TgR9YK3WEbe7CgoYoKOEwIwgYUuPRtQQaDTdwO6sQ5hHRvn+MIAdJWKYtThkJsmf4lWm35rEnrZk3XU2t0ikHDIeUYouyR3bAGe2C37Jm9/1qr7tVoejmgWW1pub0bPB7Jv/2rqtAssP+p+tOzQBEpz6tO3m2Pad5Ca+lrh6eN/OxqtD7BrtgL+b9kT+yebmDWXrXrHF89+8OPSl7oxahB8vd2/ATr8Zg8E0vkpiOZRb9VPRjFOCapH0lksIwsClTfxgnOcSGFpaQ0L2VaqVKbrxnGl5BWPgDiqZJ1</latexit>
12892 + 9837 =?
<latexit sha1_base64="3cCCvxejuSwhH3gAGZntIRJA/Ck=">AAAChHichVHNLgNRFP46/uunxUZi02iIhTRnWtWOhTRsLFsUCSIz42LS6cxkZtqExguwJRZWJBbiATyAjRew8AhiSWJj4cy0IhY4N/fec79zvnO/e4/mmIbnEz1FpLb2js6u7p5ob1//QCw+OLTq2TVXF2XdNm13XVM9YRqWKPuGb4p1xxVqVTPFmlZZCOJrdeF6hm2t+AeO2Kqqe5axa+iqz1ApvR1PUkrJZ7OZfIJSREpaybGjKIqckxMyI4El0bKiHb/DJnZgQ0cNVQhY8Nk3ocLjsQEZBIexLTQYc9kzwrjAEaLMrXGW4AyV0Qqve3zaaKEWn4OaXsjW+RaTp8vMBMbpkW7olR7olp7p49dajbBGoOWAd63JFc527Hhk+f1fVpV3H/vfrD81+9hFPtRqsHYnRIJX6E1+/fD8dXl2abwxQVf0wvov6Ynu+QVW/U2/Lomliz/0aKyFf4wb9NWFxO/Oajolz6QypelkYb7Vqm6MYgyT3I8cClhEEWWuL3CCU5xJndKUlJGyzVQp0uIM44dJc58RFJAW</latexit>
2
<latexit sha1_base64="dtxzgykiOKrKKToFg2z7LT9XiS0=">AAAChHichVHNLgNRFP6Mv/ovNhKbRkMspDnTqnYsRNhYaikSRGbGbU1MZyYz0ybVeAG2xMKKxEI8gAew8QIWfQSxJLGxcGZaEQucm3vvud8537nfvUdzTMPziRptUntHZ1d3pKe3r39gcCg6PLLh2RVXFwXdNm13S1M9YRqWKPiGb4otxxVqWTPFpna4HMQ3q8L1DNta92uO2C2rJcsoGrrqM5SjvWicEko2nU5lY5QgUpJKhh1FUeSMHJMZCSyOlq3a0XvsYB82dFRQhoAFn30TKjwe25BBcBjbRZ0xlz0jjAsco5e5Fc4SnKEyeshriU/bLdTic1DTC9k632LydJkZwyQ90S290iPd0TN9/FqrHtYItNR415pc4ewNnYytvf/LKvPu4+Cb9admH0VkQ60Ga3dCJHiF3uRXjy5e1+bzk/UpuqYX1n9FDXrgF1jVN/0mJ/KXf+jRWAv/GDfoqwux352NZEKeS6Rys/HFpVarIhjHBKa5HxksYgWrKHB9gVOc4VzqkmaklJRupkptLc4ofpi08AkM1JAU</latexit>
0
<latexit sha1_base64="P9nwiNSxLZKIKI1oebJys0uGCoU=">AAACiHichVHLLgRBFD3ae7wGG4nNxIRYTW63R09bydhYjscgQaS7FSp6ujvdPZMw8QM2logViYX4AB9g4wcsfIJYkthYuN0zIha4laq6de49t07VtXxHhhHRU5PS3NLa1t7Rmerq7untS/cPrIReJbBFyfYcL1izzFA40hWlSEaOWPMDYZYtR6xa+3NxfLUqglB67nJ04IvNsrnryh1pmxFDJU3TNWMrnaWckZ+amshnKEdkaIbOjmEYqq5mVEZiy6JhRS99hw1sw4ONCsoQcBGx78BEyGMdKgg+Y5uoMRawJ5O4wBFSzK1wluAMk9F9Xnf5tN5AXT7HNcOEbfMtDs+AmRmM0iPd0Cs90C0908evtWpJjVjLAe9WnSv8rb7joaX3f1ll3iPsfbP+1BxhB/lEq2TtfoLEr7Dr/Orh6evSzOJobYyu6IX1X9IT3fML3Oqbfb0gFi/+0GOxFv4xbtBXFzK/OytaTp3OTSxMZmcLjVZ1YBgjGOd+6JjFPIoocX2JE5zhXEkppOiKUU9VmhqcQfwwpfAJa7yREg==</latexit>
22729
train
test
(Perfect Answers!!)
26. / 35
14
Designing relevant âinductive biasesâ
âInductive biasesâ (often unintendedly) deïŹnes the interspace of instances, and then
also deïŹnes the prediction for unseen areas (crucial for out-of-distribution predictions).
Neural Networks (ReLU)
Random Forest
Linear Regression Extra Trees (no bootstrap)
Carefully designed inputs
(input only conïŹdently relevant
info into ML)
Conservative models + Strong
inductive biases that best ïŹt to
the given problem
General models having a large number
of parameters + Generalizable
inductive biases
(enables zero-shot/few-shot transfer?)
Kitchen-sink inputs (input all
potentially relevant info into ML)
Use any âphysics-informedâ conditions to further constrain or regularize
the model space sounds a good idea indeed
Complex
Simple
Small Large
Input-Output Correlation for Target
Data Required To Make ML Work
27. / 35
15
A recent news: OGB Large-Scale Challenge @ KDDCup 2021
Current ML is too data-hungry (and purportedly vulnerable to any data bias)
Gaps between technical interests and reality
Modern ML can learn any input-output mappings in theory, but more data is needed
when the input-output correlation is weak.
PCQM4M-LSC predicting DFT-calculated HOMO-LUMO energy gap of molecules given their
2D molecular graphs. (3,803,453 graphs from PubChemQC; cf. 133,885 graphs for QM9)
1st place: 10 GNNs (12-Layer Graphormer) + 8 ExpC*s (5-Layer ExpandingConv)
73 GNNs (11-Layer LiteGEMConv with Self-Supervised Pretraining)
20 GNNs (32-Layer GN with Noisy Nodes)
Test MAE 0.1200 (eV)
2nd place: Test MAE 0.1204 (eV)
3rd place: Test MAE 0.1205 (eV)
Our reality. Practical âopen-endâ cases
we can only have very limited data relative to the astronomically vast search space.
Our technical interests. weâre very excited to explore ML over large data (for pretraining +
transfer) with generalizable modular structures: CNNs vs Transformers vs GNNs vs MLPs
28. / 35
16
Todayâs talk
Gas-phase reactions on sold-phase catalyst surface (Heterogeneous catalysis)
Industrial Synthesis (e.g. Haber-Bosch), Automobile Exhaust Gas PuriïŹcation, Methane Conversion, etc.
Reactants
(Gas)
Catalysts
(Solid)
Nano-particle
surface
High Temperature, High Pressure
Adsorption
Diffusion
Dissociation
Recombination
Desorption
God made the bulk;
the surface was invented by the devil
Devilishly complex too-many-factor process!!
ââ Wolfgang Pauli
Our struggles for better ML practices with underspeciïŹed, sparse, biased
observational data (i.e. a collection of experimental facts from literature)
29. / 35
17
The base dataset
http://www.fhi-berlin.mpg.de/acnew/department/pages/ocmdata.html
https://www.nature.com/articles/s41467-019-08325-8#Sec19
Oxidative coupling of methane (OCM) reactions
Methane (CH4) is partially oxidized to C2 hydrocarbons such as ethane (C2H6) and ethylene (C2H4) in a single step
Elemental composition of catalyst (mol%) Process parameters + Preparation Catalytic performance
⹠Zavyalova, U.; Holena, M.; Schlögl, R.; Baerns, ChemCatChem 2011.
âą Followup:
Kondratenko, E. V.; SchlĂŒter, M.; Baerns, M.; Linke, D.; Holena, M.
Catal. Sci. Technol. 2015.
âą Renalysis with Corrections & Outlier Removal
Schmack, R.; Friedrich, A.; Kondratenko, E. V.; Polte, J.; Werwatz, A.;
Kraehnert, R. Nat Commun 2019.
1866 catalyst records from 421 reports
30. / 35
18
Problem #1: Itâs underspecified
âą Each catalyst was mostly measured at different reaction conditions.
âą Only a few are measured under multiple conditions
158 compositions > 2 conditions
60 compositions > 3 conditions
26 compositions > 4 conditions
La:100.0 x 24
Mg:100.0 x 18
Ca:100.0 x 18
Sm:100.0 x 13
Nd:100.0 x 10
Ba:100.0 x 9
Y:100.0 x 9
Ce:100.0 x 9
Zr:100.0 x 9
Sr:100.0 x 7
Si:100.0 x 7
Gd:100.0 x 7
Na:8.9 Si:83.1 Mn:3.5 W:4.5 x 7
Pr:100.0 x 6
Eu:100.0 x 6
Yb:100.0 x 5
Al:100.0 x 4
Li:10.0 Mg:90.0 x 4
Mg:90.9 La:9.1 x 4
Li:9.1 Mg:90.9 x 4
Tb:100.0 x 4
Li:20.0 Cl:20.0 Zr:60.0 x 4
Na:20.0 Cl:20.0 Zr:60.0 x 4
Cl:20.0 K:20.0 Zr:60.0 x 4
Cl:20.0 Rb:20.0 Zr:60.0 x 4
Cl:20.0 Zr:60.0 Cs:20.0 x 4
âą No replicates in the same conditions
âą But as we see later, reaction conditions are quite inïŹuential. Because of
this, âno generally valid correlation between a catalystâs composition,
its structure and its OCM performance has been established yet.â
Strong limitation of observational data (just passively acquired)
Observational study Interventional study
32. / 35
20
Problem #3: Itâs biased
Unavoidable Human-Caused Biases
âmost chemical experiments are planned by human scientists and therefore are subject
to a variety of human cognitive biases, heuristics and social inïŹuences.â
Jia, X.; Lynch, A.; Huang, Y.; Danielson, M.; Langâat, I.; Milder, A.; Ruby, A. E.; Wang, H.; Friedler, S. A.;
Norquist, A. J.; Schrier, J. Nature 2019, 573 (7773), 251â255.
Catalyst such as LaO3, Li/MgO, and
Mn/Na2WO4/SiO2 extensively studied.
33. / 35
21
Our solutions
Solution to Problem #1 (UnderspeciïŹcation)
Tree ensemble regressors with prediction variances are used to make robust and less
risky prediction as well as to quantify how uncertain each ML prediction is.
Solution to Problem #2 (Sparsity)
The catalyst representation called SWED (Sorted Weighted Elemental Descriptors) is
developed to represent catalysts not in a one-hot fashion but by elemental descriptors.
Solution to Problem #3 (Strong Bias)
On the top of the above two, sequential model-based optimization with SWED only by 3
descriptors (electronegativity, density, and ÎHfus) as well as 8 descriptors are explored.
Also, for suggested candidates to be worth testing, SHAP interpretations are provided.
OCM Dataset Update, Reanalysis, Exploration
The original dataset (1866 catalyst records from 421 reports until 2009) is extended to
4559 catalyst records from 542 reports from 2010 to 2019, and reanalyzed.
34. / 35
22
#1. Tree ensemble regression with uncertainty
Tree ensemble regressors with prediction variances are used to make robust and less
risky prediction as well as to quantify how uncertain each ML prediction is.
Gradient Boosted Trees
Extra Trees (no bootstrap)
Random Forest
Extra Trees (bootstrap)
sample max
sample min
GradientBoostingRegressor
LGBMRegressor
RandomForestRegressor
ExtraTreesRegressor
By quantile
regression
to .16, .5, .84
quantiles
Naturally by
the law of
total variance
bounded
prediction
Avoid the risk of unintended extrapolation?
(High-dimensional feature spaces can be
counterintuitiveâŠ)
35. / 35
23
#2. SWED representation of catalysts
The catalyst representation called SWED (Sorted Weighted Elemental Descriptors) is
developed to represent catalysts not in a one-hot fashion but by elemental descriptors.
Key Idea
one-hot-like features below are statistically incomparable
so represent catalysts instead by any elemental descriptors
to represent arbitrary (used/unused) elements in a common
ground, considering chemical similarities.
36. / 35
24
SWED (Sorted Weighted Elemental Descriptors)
âą Product terms can represent interaction effects between variables (e.g. probabilistic gating, attention, âŠ)
and furthermore, they can zero out the feature when the corresponding element is 0%.
âą Sorted concatenation is lossless, and was better than weighted sum or weighted max.
âą SWED lose the exact composition. To compensate, we also developed a SWEDâcomposition estimator.
âą We tried many other things (matrix decomposition, Aitchison geometry, GNN, etc) that didnât work.
+
37. / 35
25
#3. Optimism in the face of uncertainty
<latexit sha1_base64="Nr2ImZk9Gvp+/1cBWPwAK6yswp0=">AAACiXichVFNLwNRFD0dX1VfxUZi02iIVXNHBOmq0Y1lP7QkiMyMh9H5ysy0UU3/gJWdYEViIX6AH2DjD1j0J4gliY2F2+kkguBO3rzzzrvnvvPeVR1D93yiVkTq6u7p7Yv2xwYGh4ZH4qNjZc+uupooabZhu+uq4glDt0TJ131DrDuuUEzVEGtqJdveX6sJ19Nta9WvO2LLVPYsfVfXFJ+p8qZqNg6b2/EkpSiIxE8ghyCJMHJ2/A6b2IENDVWYELDgMzagwONvAzIIDnNbaDDnMtKDfYEmYqytcpbgDIXZCv/3eLURshav2zW9QK3xKQYPl5UJTNMj3dALPdAtPdH7r7UaQY22lzrPakcrnO2R44ni278qk2cf+5+qPz372MVS4FVn707AtG+hdfS1o9OXYrow3ZihK3pm/5fUonu+gVV71a7zonDxhx+VvfCLcYPk7+34CcpzKXkhRfn5ZGY5bFUUk5jCLPdjERmsIIcS1z/ACc5wLg1IsrQkpTupUiTUjONLSNkPIVKSSQ==</latexit>
x
<latexit sha1_base64="Qqfga3YaBoUI0Yb7fQVzumPP1Sw=">AAAChHichVHLSsNAFD1GrbW+qm4EN8WiuJBy6xsXUnTjsg9rC1VKEqc1mCYhSQu1+AO6VVy4UnAhfoAf4MYfcNFPEJcKblx4mwZEi3rDZM6cuefOmbmKpWuOS9Tskrp7egN9wf7QwODQ8Eh4dGzHMau2KrKqqZt2XpEdoWuGyLqaq4u8ZQu5ougipxxutvZzNWE7mmlsu3VL7FXksqGVNFV2mUrVi+EoxciLSCeI+yAKP5Jm+B672IcJFVVUIGDAZaxDhsNfAXEQLOb20GDOZqR5+wLHCLG2ylmCM2RmD/lf5lXBZw1et2o6nlrlU3QeNisjmKYnuqVXeqQ7eqaPX2s1vBotL3WelbZWWMWRk4nM+7+qCs8uDr5Uf3p2UcKq51Vj75bHtG6htvW1o4vXzFp6ujFD1/TC/q+oSQ98A6P2pt6kRPryDz8Ke+EX4wbFf7ajE+zMx+LLMUotRhMbfquCmMQUZrkfK0hgC0lkub7AKc5wLgWkOWlBWmqnSl2+ZhzfQlr/BNcbj/U=</latexit>
y
sample max
sample min
We would like to ïŹnd X better than (hopefully) the currently known best .
<latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit>
x1
<latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit>
x2
<latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit>
?
<latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit>
?
<latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit>
?
<latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit>
?
<latexit sha1_base64="qj6YHRPaHOekeYfY3235sFbWBxQ=">AAAChHichVHNLgNRFP6M/98WG4lNoyEW0pzpn9YCYWOpqDYpkZlxMel0ZjIzbULjBdgSCysSC/EAHsDGC1h4BLEksbFwZloRi3Ju7r3nfud85373HtU2dNcjem6T2js6u7p7evv6BwaHQuHhkU3XqjqayGuWYTlFVXGFoZsi7+meIYq2I5SKaoiCWl7244WacFzdMje8Q1tsV5R9U9/TNcVjKLewE45SLJtJUzIeoRhRJk5pdlIkZ+VsRGbEtyiatmqF77GFXVjQUEUFAiY89g0ocHmUIINgM7aNOmMOe3oQFzhGH3OrnCU4Q2G0zOs+n0pN1OSzX9MN2BrfYvB0mBnBJD3RLb3RI93RC322rFUPavhaDnlXG1xh74ROxtY//mVVePdw8MP6U7OHPWQCrTprtwPEf4XW4NeOLt7W59Ym61N0Ta+s/4qe6YFfYNbetZucWLv8Q4/KWvjHuEHfXYi0djbjMTkdS+SS0cWlZqt6MI4JTHM/ZrGIFawiz/UFTnGGc6lLmpESUqqRKrU1OaP4ZdL8F+zrkAQ=</latexit>
?
What next location of is likely to give higher ?
<latexit sha1_base64="GizvUUdXn2Kx+/exfuoNL7eSBEo=">AAACiXichVG7TgJBFD2sL8QHqI2JDZFgrMgsEh5WRBtLHgImYMzuOuDqvrK7EJX4A1Z2Rq00sTB+gB9g4w9Y8AnGEhMbCy8LxliodzM7Z87cc+fMXNnSVMdlrOMThoZHRsf844GJyanpYGhmtuyYTVvhJcXUTHtLlhyuqQYvuaqr8S3L5pIua7wiH6z39istbjuqaWy6Rxbf1qWGodZVRXKJKtdkvX14shOKsFg8kxRTK2EPJFKiB+JiOhMWY8yLCAaRM0MPqGEXJhQ0oYPDgEtYgwSHvipEMFjEbaNNnE1I9fY5ThAgbZOyOGVIxB7Qv0Gr6oA1aN2r6XhqhU7RaNikDCPKntkd67Inds9e2MevtdpejZ6XI5rlvpZbO8HT+eL7vyqdZhd736o/PbuoI+15Vcm75TG9Wyh9fev4vFtcLUTbS+yGvZL/a9Zhj3QDo/Wm3OZ54eoPPzJ5oRejBn11Ifw7KMdjYjK2kk9EsmuDVvmxgEUsUz9SyGIDOZSo/j7OcIFLYUIQhbSw2k8VfAPNHH6EsP4J1yOSpA==</latexit>
x <latexit sha1_base64="fNJj/X2guXDQcypavSpjd2rZHL0=">AAAChHichVG7SgNBFD1ZXzG+ojaCzWJQLGSZTWISLUS0sczDqKAiu+skLu6L3U0gBn9AW8XCSsFC/AA/wMYfsMgniKWCjYV3NxGxUO8yO2fO3HPnzFzVMXTPZ6wVEbq6e3r7ov2xgcGh4ZH46NiGZ9dcjZc127DdLVXxuKFbvOzrvsG3HJcrpmrwTfVwNdjfrHPX021r3W84fNdUqpZe0TXFJ6rQ2IsnmJRcyMjZlBiCdFYOQVLOLYiyxMJIoBN5O36PHezDhoYaTHBY8AkbUODRtw0ZDA5xu2gS5xLSw32OY8RIW6MsThkKsYf0r9Jqu8NatA5qeqFao1MMGi4pRUyzJ3bLXtkju2PP7OPXWs2wRuClQbPa1nJnb+RkovT+r8qk2cfBt+pPzz4qyIVedfLuhExwC62trx9dvJYWi9PNGXbNXsj/FWuxB7qBVX/Tbgq8ePmHH5W80ItRg766IP4ONpKSnJFShXRieaXTqigmMYVZ6kcWy1hDHmWqz3GKM5wLvcKckBLm26lCpKMZx48Qlj4BjPuQUA==</latexit>
y
Exploitation Make the best decision given current information
Exploration Gather more information by probing uncertain areas
A fundamental choice: exploitation-exploration tradeoff
<latexit sha1_base64="g1U7Tzgl+Lj3TSOL2Mnl/fuxw90=">AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxVW5UVEpLooiuOzDPqAtJYljDc2LJC3W0B9w6cZF3Si4ED/AD3DjD7joJ4jLCm5ceJsGRIv1hsmcOXPPnTNzRUORLRux6+PGxicmp/zTgZnZ4FwoPL+Qt/SGKbGcpCu6WRQFiymyxnK2bCusaJhMUEWFFcT6fn+/0GSmJevakd0yWEUVapp8IkuCTVSxLKrOWbvKV8NRjKMbkWHAeyAKXqT08COU4Rh0kKABKjDQwCasgAAWfSXgAcEgrgIOcSYh2d1n0IYAaRuUxShDILZO/xqtSh6r0bpf03LVEp2i0DBJGYEYvuA99vAZH/AVP/+s5bg1+l5aNIsDLTOqoYul7Me/KpVmG06/VSM923ACO65XmbwbLtO/hTTQN8+vetlEJuas4i2+kf8b7OIT3UBrvkt3aZbpjPAjkhd6MWoQ/7sdwyC/Hue34pjejCb3vFb5YRlWYI36sQ1JOIQU5Nw+XEIHrrkgt8EluN1BKufzNIvwI7iDL6QMku0=</latexit>
x1
<latexit sha1_base64="HrABz/ySZPgA1Ybj81k3XV8tH68=">AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaVEgLKYKW/uQPqMjMdLXB+WNmlGzwBVq2aWGbghbRA/QAbXqBFj5CtDRo06LjOBAl2Rnu3O9+93znfvceQZcl00LsebiJyanpGe+sb27evxAILi7lTa1piCwnarJmFAXeZLKkspwlWTIr6gbjFUFmBaFxMNgvtJhhSpp6bLV1VlH4uirVJJG3iCqWBcU+61Tj1WAYo+hEaBTEXBAGN1Ja8BHKcAIaiNAEBRioYBGWgQeTvhLEAEEnrgI2cQYhydln0AEfaZuUxSiDJ7ZB/zqtSi6r0npQ03TUIp0i0zBIGYIIvuA99vEZH/AVP/+sZTs1Bl7aNAtDLdOrgYuV7Me/KoVmC06/VWM9W1CDXcerRN51hxncQhzqW+dX/WwiE7HX8RbfyP8N9vCJbqC23sW7NMt0x/gRyAu9GDUo9rsdoyAfj8a2o5jeCif33VZ5YRXWYIP6sQNJOIIU5Jw+XEIXrjk/t8kluL1hKudxNcvwI7jDL6Ysku4=</latexit>
x2
exploitative choice
explorative choice
<latexit sha1_base64="TEyxI6Ffe+b+OEFRulxYJ0sUB8w=">AAAChnichVHLTsJAFD3UF+ID1I2JGyLBGBdk8IVxRXTjkoc8EkTS1gEbStu0hQSJP2DiVhauNHFh/AA/wI0/4IJPMC4xcePCS2lilIi3mc6ZM/fcOTNXMlTFshnreISR0bHxCe+kb2p6ZtYfmJvPWnrdlHlG1lXdzEuixVVF4xlbsVWeN0wu1iSV56Tqfm8/1+Cmpejaod00eLEmVjSlrMiiTVS6ebxWCoRYhDkRHARRF4TgRkIPPOIIJ9Aho44aODTYhFWIsOgrIAoGg7giWsSZhBRnn+McPtLWKYtThkhslf4VWhVcVqN1r6blqGU6RaVhkjKIMHth96zLntkDe2Wff9ZqOTV6Xpo0S30tN0r+i8X0x7+qGs02Tr9VQz3bKGPH8aqQd8NhereQ+/rGWbub3k2FWyvslr2R/xvWYU90A63xLt8leep6iB+JvNCLUYOiv9sxCLLrkeh2ZCO5GYrvua3yYgnLWKV+xBDHARLIUP0KLnGFtuAVIsKWEOunCh5Xs4AfIcS/AEWBkJQ=</latexit>
yâ€
<latexit sha1_base64="ivVWRQUyNt8f9PjHjZnGRvohU5E=">AAAChnichVG5TsNAEH2YK9wBGiSaiCgIUURjbqgiaChDQg6JS7ZZgoVjW7YTKUT8ABItFFQgUSA+gA+g4Qco8gmIMkg0FEwcI6BImNXuzr6ZN/t2R7UN3fWIah1SZ1d3T2+or39gcGh4JDw6lnWtkqOJjGYZlpNXFVcYuikynu4ZIm87QimqhsipJxuNeK4sHFe3zG2vYou9olIw9SNdUzyG0pX92YNwlOLkW+SXs0jy6pIckQMkisCSVvgRuziEBQ0lFCFgwmPfgAKXxw5kEGzG9lBlzGFP9+MCZ+hnbomzBGcojJ7wWuDTToCafG7UdH22xrcYPB1mRhCjF7qnOj3TA73SZ8taVb9GQ0uFd7XJFfbByPlE+uNfVpF3D8c/rLaaPRxhxdeqs3bbRxqv0Jr88ulVPb2WilWn6ZbeWP8N1eiJX2CW37W7LZG6bqNHZS38Y9yg7y5EWjvZubi8FJ/fWogm1oNWhTCJKcxwP5aRwCaSyHD9Ai5wiSspJMWlRWm5mSp1BJxx/DEp8QVx95Cq</latexit>
yâ€
Random choice (e.g. random design) or
evenly spaced sampling (e.g. full factorial
design) can also work for lower dimensional
exploration.