Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2020 KNIME AG. All Right Reserved.
Codeless Deep Learning for Language Modeling
and Image Classification
Corey Weisinger...
© 2020 KNIME AG. All Rights Reserved.
Agenda
• Introduction to the open source tool KNIME Analytics Platform
• Introductio...
© 2020 KNIME AG. All Rights Reserved.
What is KNIME Analytics Platform?
• A tool for data analysis, manipulation, visualiz...
© 2020 KNIME AG. All Rights Reserved.
Visual KNIME Workflows
NODES perform tasks on data
Nodes are combined to create
WORK...
© 2020 KNIME AG. All Rights Reserved.
Analysis & Mining
Statistics
Data Mining
Machine Learning
Web Analytics
Text Mining
...
© 2020 KNIME AG. All Rights Reserved.
KNIME Software - Better Decision-making, Faster!
DeployManageAutomateCollaborate
Ope...
© 2020 KNIME AG. All Rights Reserved. 7
Introduction to RNN and their Applications
© 2020 KNIME AG. All Rights Reserved. 8
Neural Network Review
© 2020 KNIME AG. All Rights Reserved.
Let’s start with a history lesson
Neuron Networks
9
Neural networks are inspired by ...
© 2020 KNIME AG. All Rights Reserved.
Fully connected, feed forward networks
10
Input
Layer
Hidden
Layer
Output
Layer Forw...
© 2020 KNIME AG. All Rights Reserved.
Frequently used activation functions
11
Sigmoid Tanh Rectified Linear Unit (ReLU)
© 2020 KNIME AG. All Rights Reserved.
Fully connected, feed forward networks- simplified
12
Input
Layer
Hidden
Layer
Outpu...
© 2020 KNIME AG. All Rights Reserved. 13
What are RNNs and LSTMs?
© 2020 KNIME AG. All Rights Reserved.
What are RNNs?
• Recurrent Neural Network (RNN) are a family of neural networks used...
© 2020 KNIME AG. All Rights Reserved.
Why do we need RNNs for sequential data?
• Goal: Translation network from German to ...
© 2020 KNIME AG. All Rights Reserved.
Why do we need RNNs for sequential data?
• Problems:
– Each time step is completely ...
© 2020 KNIME AG. All Rights Reserved.
What are RNNs?
17
Image Source: Christopher Olah, https://colah.github.io/posts/2015...
© 2020 KNIME AG. All Rights Reserved.
From feed forward to recurrent neural networks
18
∑
∑
∑
∑
∑
∑
∑
𝒙
𝟐
𝒚
𝟑
𝒙
𝟐
𝒚
𝟑
© 2020 KNIME AG. All Rights Reserved.
From feed forward to recurrent neural networks
19
∑
∑
∑
𝒙
𝟐
𝒚
𝟑
∑
∑
∑
𝒙
𝟐
𝒚
𝟑
∑
∑
∑
...
© 2020 KNIME AG. All Rights Reserved.
Simple RNN unit
20
Image Source: Christopher Olah, https://colah.github.io/posts/201...
© 2020 KNIME AG. All Rights Reserved.
Limitations of simple layer structures
21
The “memory” of simple RNNs is sometimes t...
© 2020 KNIME AG. All Rights Reserved.
LSTM = Long Short Term Memory Unit
• Special type of unit with three gates
– Forget ...
© 2020 KNIME AG. All Rights Reserved.
Different network-structures and applications
Many to Many
23
A A A
<sos>
A
like sai...
© 2020 KNIME AG. All Rights Reserved.
Different network-structures and applications
Many to one
24
A A A A
I
A
like to go ...
© 2020 KNIME AG. All Rights Reserved.
How can we train a recurrent neural network for text generation
• Decide between cha...
© 2020 KNIME AG. All Rights Reserved. 26
The Creative Side of AI: Naming New Products
© 2020 KNIME AG. All Rights Reserved.
The Product Naming Phase
Naming new products is not as easy as it might sound …
27
W...
© 2020 KNIME AG. All Rights Reserved.
The Case Study: new line of outdoor clothing
Outdoor or hiking clothing line needs
n...
© 2020 KNIME AG. All Rights Reserved.
Let’s build a model to generate fictional mountain names!
29
© 2020 KNIME AG. All Rights Reserved.
Quick sneak preview – Find the fake mountain names
30
Set 1
Barlock Mountain
Casterl...
© 2020 KNIME AG. All Rights Reserved.
The Dataset
31
• 33,012 names of mountains in
the US
• Extracted from Wikipedia via ...
© 2020 KNIME AG. All Rights Reserved.
Many to many structure
32
© 2020 KNIME AG. All Rights Reserved.
LSTM based Neural Network: Many to Many
33
0,1,0,0,0,0,0,0,0,...,0
0,0,0,1,0,0,0,0,0...
© 2020 KNIME AG. All Rights Reserved.
The KNIME Keras Integration
34
• Codeless GUI based
• Fully opensource
• Keras funct...
© 2020 KNIME AG. All Rights Reserved.
Codeless?
35
LSTMstates
© 2020 KNIME AG. All Rights Reserved.
Model training workflow
• Read in and transform the input
dataset
• Define the struc...
© 2020 KNIME AG. All Rights Reserved.
Now that we have a model, how do we use it?
37
© 2020 KNIME AG. All Rights Reserved.
Neural network: Code-free
Layers of our neural network:
• Input layer of size [?, 95...
© 2020 KNIME AG. All Rights Reserved.
Want to see the names?
39
Advantages:
- They remind you of mountains. Do
not they?
-...
© 2020 KNIME AG. All Rights Reserved. 40
Yo! AI Generated Rap Songs
© 2020 KNIME AG. All Rights Reserved.
Bots & speaking styles
41
- Articulate
- Chatty
- Clean
- Conversational
- Crisp
- D...
© 2020 KNIME AG. All Rights Reserved.
How should my bot speak?
• I want my bot to answer in a polite tone in some situatio...
© 2020 KNIME AG. All Rights Reserved.
Let‘s start with impolite ...
• Let‘s generate a rap song!
• This is a free text gen...
© 2020 KNIME AG. All Rights Reserved.
On the KNIME Hub
44
https://hub.knime.com
© 2020 KNIME AG. All Rights Reserved.
On the KNIME Hub
45
https://kni.me/w/mGO9nXhmjzIKiqHU
© 2020 KNIME AG. All Rights Reserved.
LSTM based Neural Network
46
N
0,1,0,0,0,0,0,0,0,...,0
0,0,0,1,0,0,0,0,0,...,0
u o h...
© 2020 KNIME AG. All Rights Reserved.
Creative AI: the training workflow
47
© 2020 KNIME AG. All Rights Reserved.
Creative AI: the deployment workflow
48
© 2020 KNIME AG. All Rights Reserved.
don't need no advice
You're not here and we both know why, so
Move from me when you'...
© 2020 KNIME AG. All Rights Reserved. 50
To be or not to be … Shakespearian text?
© 2020 KNIME AG. All Rights Reserved.
Now for the poetic text ...
• How would Shakespeare say it?
• I can reuse the same L...
© 2020 KNIME AG. All Rights Reserved.
Want to see some AI/Shakespeare text?
To see you here before me. O my soul’s joy!
I ...
© 2020 KNIME AG. All Rights Reserved. 53
Conclusions
© 2020 KNIME AG. All Rights Reserved.
Can AI be creative?
Yo!
This post is about generating free text
with a deep learning...
© 2020 KNIME AG. All Rights Reserved.
References
• https://www.datanami.com/2019/04/04/product-naming-with-deep-learning/
...
© 2020 KNIME AG. All Rights Reserved.
Free Book as a Thank You
Free Copy of e-book:
“Practicing Data Science.
A Collection...
© 2019 KNIME AG. All Rights Reserved.
The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by
KNIME A...
© 2020 KNIME AG. All Right Reserved.
Transfer Learning
Corey Weisinger
© 2020 KNIME AG. All Rights Reserved.
What is Transfer Learning?
• Transfer learning can be defined
as the attempt to util...
© 2020 KNIME AG. All Rights Reserved.
Transfer Learning – Basic Example
Input Data
Sepal Length
Sepal Width
Petal Length
P...
© 2020 KNIME AG. All Rights Reserved.
Cancer Cell Classification Use Case
• Uses Keras Deep Learning
model.
• Learning Tra...
© 2020 KNIME AG. All Rights Reserved.
Why is it helpful?
• Deep Learning requires tons of
data
• Image classification requ...
© 2020 KNIME AG. All Rights Reserved.
Feature Extraction in Neural Networks
• One of the benefits of
deep learning is its ...
© 2020 KNIME AG. All Rights Reserved.
Convolutional Layer
• Instead of connecting every
neuron to the new layer a
sliding ...
© 2020 KNIME AG. All Rights Reserved.
Convolutional Layer Cont.
• Stride:
– Distance between adjacent
windows (windows can...
© 2020 KNIME AG. All Rights Reserved.
VGG16 Model
• Several blocks of small convolution layers followed by max pooling
lay...
© 2020 KNIME AG. All Rights Reserved.
Everything in KNIME
1. Download Data Set
2. Pre-Process Data Set
1. Crop, Normalize,...
© 2020 KNIME AG. All Rights Reserved.
Pre-Processing
• Input size
– VGG16: 3x64x64 image
– Us: 1388x1040x3 image
• Image V...
© 2020 KNIME AG. All Rights Reserved.
Configuring the Network
• Load VGG16 Network
• Add custom layers to produce 3 class ...
© 2020 KNIME AG. All Rights Reserved.
Training the Network
• We train the newly
configured network for 25
epochs
• Only up...
© 2020 KNIME AG. All Rights Reserved.
Post-Processing
• Combine image patches
– Average prediction by class
– Use max prob...
© 2020 KNIME AG. All Rights Reserved.
Finding the Example
© 2020 KNIME AG. All Rights Reserved.
Related KNIME Blog Articles
• Original use case, involves
some coding:
– https://www...
© 2020 KNIME AG. All Rights Reserved.
Citations
• Convolution Layer: https://towardsdatascience.com/a-
comprehensive-guide...
© 2020 KNIME AG. All Rights Reserved.
KNIME Course Books
Course books downloadable from
KNIME Press
https://www.knime.com/...
© 2019 KNIME AG. All Rights Reserved.
The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by
KNIME A...
© 2020 KNIME AG. All Rights Reserved.
Next Meetup
• Planning for April, then afterward June
• Next location: IBM campus on...
Upcoming SlideShare
Loading in …5
×

of

Codeless Deep Learning for Language Modeling and Image Classification Slide 1 Codeless Deep Learning for Language Modeling and Image Classification Slide 2 Codeless Deep Learning for Language Modeling and Image Classification Slide 3 Codeless Deep Learning for Language Modeling and Image Classification Slide 4 Codeless Deep Learning for Language Modeling and Image Classification Slide 5 Codeless Deep Learning for Language Modeling and Image Classification Slide 6 Codeless Deep Learning for Language Modeling and Image Classification Slide 7 Codeless Deep Learning for Language Modeling and Image Classification Slide 8 Codeless Deep Learning for Language Modeling and Image Classification Slide 9 Codeless Deep Learning for Language Modeling and Image Classification Slide 10 Codeless Deep Learning for Language Modeling and Image Classification Slide 11 Codeless Deep Learning for Language Modeling and Image Classification Slide 12 Codeless Deep Learning for Language Modeling and Image Classification Slide 13 Codeless Deep Learning for Language Modeling and Image Classification Slide 14 Codeless Deep Learning for Language Modeling and Image Classification Slide 15 Codeless Deep Learning for Language Modeling and Image Classification Slide 16 Codeless Deep Learning for Language Modeling and Image Classification Slide 17 Codeless Deep Learning for Language Modeling and Image Classification Slide 18 Codeless Deep Learning for Language Modeling and Image Classification Slide 19 Codeless Deep Learning for Language Modeling and Image Classification Slide 20 Codeless Deep Learning for Language Modeling and Image Classification Slide 21 Codeless Deep Learning for Language Modeling and Image Classification Slide 22 Codeless Deep Learning for Language Modeling and Image Classification Slide 23 Codeless Deep Learning for Language Modeling and Image Classification Slide 24 Codeless Deep Learning for Language Modeling and Image Classification Slide 25 Codeless Deep Learning for Language Modeling and Image Classification Slide 26 Codeless Deep Learning for Language Modeling and Image Classification Slide 27 Codeless Deep Learning for Language Modeling and Image Classification Slide 28 Codeless Deep Learning for Language Modeling and Image Classification Slide 29 Codeless Deep Learning for Language Modeling and Image Classification Slide 30 Codeless Deep Learning for Language Modeling and Image Classification Slide 31 Codeless Deep Learning for Language Modeling and Image Classification Slide 32 Codeless Deep Learning for Language Modeling and Image Classification Slide 33 Codeless Deep Learning for Language Modeling and Image Classification Slide 34 Codeless Deep Learning for Language Modeling and Image Classification Slide 35 Codeless Deep Learning for Language Modeling and Image Classification Slide 36 Codeless Deep Learning for Language Modeling and Image Classification Slide 37 Codeless Deep Learning for Language Modeling and Image Classification Slide 38 Codeless Deep Learning for Language Modeling and Image Classification Slide 39 Codeless Deep Learning for Language Modeling and Image Classification Slide 40 Codeless Deep Learning for Language Modeling and Image Classification Slide 41 Codeless Deep Learning for Language Modeling and Image Classification Slide 42 Codeless Deep Learning for Language Modeling and Image Classification Slide 43 Codeless Deep Learning for Language Modeling and Image Classification Slide 44 Codeless Deep Learning for Language Modeling and Image Classification Slide 45 Codeless Deep Learning for Language Modeling and Image Classification Slide 46 Codeless Deep Learning for Language Modeling and Image Classification Slide 47 Codeless Deep Learning for Language Modeling and Image Classification Slide 48 Codeless Deep Learning for Language Modeling and Image Classification Slide 49 Codeless Deep Learning for Language Modeling and Image Classification Slide 50 Codeless Deep Learning for Language Modeling and Image Classification Slide 51 Codeless Deep Learning for Language Modeling and Image Classification Slide 52 Codeless Deep Learning for Language Modeling and Image Classification Slide 53 Codeless Deep Learning for Language Modeling and Image Classification Slide 54 Codeless Deep Learning for Language Modeling and Image Classification Slide 55 Codeless Deep Learning for Language Modeling and Image Classification Slide 56 Codeless Deep Learning for Language Modeling and Image Classification Slide 57 Codeless Deep Learning for Language Modeling and Image Classification Slide 58 Codeless Deep Learning for Language Modeling and Image Classification Slide 59 Codeless Deep Learning for Language Modeling and Image Classification Slide 60 Codeless Deep Learning for Language Modeling and Image Classification Slide 61 Codeless Deep Learning for Language Modeling and Image Classification Slide 62 Codeless Deep Learning for Language Modeling and Image Classification Slide 63 Codeless Deep Learning for Language Modeling and Image Classification Slide 64 Codeless Deep Learning for Language Modeling and Image Classification Slide 65 Codeless Deep Learning for Language Modeling and Image Classification Slide 66 Codeless Deep Learning for Language Modeling and Image Classification Slide 67 Codeless Deep Learning for Language Modeling and Image Classification Slide 68 Codeless Deep Learning for Language Modeling and Image Classification Slide 69 Codeless Deep Learning for Language Modeling and Image Classification Slide 70 Codeless Deep Learning for Language Modeling and Image Classification Slide 71 Codeless Deep Learning for Language Modeling and Image Classification Slide 72 Codeless Deep Learning for Language Modeling and Image Classification Slide 73 Codeless Deep Learning for Language Modeling and Image Classification Slide 74 Codeless Deep Learning for Language Modeling and Image Classification Slide 75 Codeless Deep Learning for Language Modeling and Image Classification Slide 76 Codeless Deep Learning for Language Modeling and Image Classification Slide 77
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1 Like

Share

Download to read offline

Codeless Deep Learning for Language Modeling and Image Classification

Download to read offline

An introduction to recurrent neural networks and LSTM units followed by some example applications for language modeling.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Codeless Deep Learning for Language Modeling and Image Classification

  1. 1. © 2020 KNIME AG. All Right Reserved. Codeless Deep Learning for Language Modeling and Image Classification Corey Weisinger Corey.Weisinger@knime.com Rosaria Silipo Rosaria.Silipo@knime.com
  2. 2. © 2020 KNIME AG. All Rights Reserved. Agenda • Introduction to the open source tool KNIME Analytics Platform • Introduction to RNN units and their Applications • Transfer learning for image classification 2
  3. 3. © 2020 KNIME AG. All Rights Reserved. What is KNIME Analytics Platform? • A tool for data analysis, manipulation, visualization, and reporting • Based on the graphical programming paradigm
  4. 4. © 2020 KNIME AG. All Rights Reserved. Visual KNIME Workflows NODES perform tasks on data Nodes are combined to create WORKFLOWS Status Inputs Outputs Not Configured Configured Executed Error
  5. 5. © 2020 KNIME AG. All Rights Reserved. Analysis & Mining Statistics Data Mining Machine Learning Web Analytics Text Mining Network Analysis Social Media Analysis R, Weka, Python Community / 3rd Data Access MySQL, Oracle, ... SAS, SPSS, ... Excel, Flat, ... Hive, Impala, ... XML, JSON, PMML Text, Doc, Image, ... Web Crawlers Industry Specific Community / 3rd Transformation Row Column Matrix Text, Image Time Series Java Python Community / 3rd Visualization R JFreeChart JavaScript Plotly Community / 3rd Deployment via BIRT PMML XML, JSON Databases Excel, Flat, etc. Text, Doc, Image Industry Specific Community / 3rd Over 2000 Native and Embedded Nodes Included:
  6. 6. © 2020 KNIME AG. All Rights Reserved. KNIME Software - Better Decision-making, Faster! DeployManageAutomateCollaborate OpenSource KNIME Server Partner Extensions KNIME Analytics Platform Community Extensions KNIME Extensions KNIME Integrations VisualizeAnalyzeTransformIntegrateLoad
  7. 7. © 2020 KNIME AG. All Rights Reserved. 7 Introduction to RNN and their Applications
  8. 8. © 2020 KNIME AG. All Rights Reserved. 8 Neural Network Review
  9. 9. © 2020 KNIME AG. All Rights Reserved. Let’s start with a history lesson Neuron Networks 9 Neural networks are inspired by biological neural networks 𝑥 𝑥 ∑ σ0,8 𝑤 𝑤 𝑏 𝑎 = 𝜎(𝑥 𝑤 + 𝑥 𝑤 + 𝑏)
  10. 10. © 2020 KNIME AG. All Rights Reserved. Fully connected, feed forward networks 10 Input Layer Hidden Layer Output Layer Forward pass: ∑ ∑ ∑ ∑ 𝑤 , 𝑤 , 𝑤 , 𝑤 , 𝑤 , 𝑏 𝑏 𝑏 𝑤 , 𝑤 , 𝑤 , 𝑤 , 𝑏
  11. 11. © 2020 KNIME AG. All Rights Reserved. Frequently used activation functions 11 Sigmoid Tanh Rectified Linear Unit (ReLU)
  12. 12. © 2020 KNIME AG. All Rights Reserved. Fully connected, feed forward networks- simplified 12 Input Layer Hidden Layer Output Layer Forward pass: ∑ ∑ ∑ ∑ 𝒙 𝟐 𝒚 𝟑
  13. 13. © 2020 KNIME AG. All Rights Reserved. 13 What are RNNs and LSTMs?
  14. 14. © 2020 KNIME AG. All Rights Reserved. What are RNNs? • Recurrent Neural Network (RNN) are a family of neural networks used for processing sequential data • RNNs are used for all sorts of tasks: – Language modeling / Text generation – Text classification – Neural machine translation – Image captioning – Speech to text – Numerical time series data, e.g. sensor data 14
  15. 15. © 2020 KNIME AG. All Rights Reserved. Why do we need RNNs for sequential data? • Goal: Translation network from German to English “Ich mag Schokolade” => “I like chocolate” • One option: Use feed forward network to translate word by word • But what about this sentence? “Kathrin mag Schokolade” => “Kathrin likes chocolate” 15 𝑥 ∑ σ 𝑦 ∑ σ ∑ σ ∑ σ ∑ σ ∑ σ ∑ σ ∑ σ Input x Output y Ich I mag like Schokolade chocolate
  16. 16. © 2020 KNIME AG. All Rights Reserved. Why do we need RNNs for sequential data? • Problems: – Each time step is completely independent – For translations we need context – More general: we need a network that remembers inputs from the past • Solution: Recurrent neural networks 16 𝑥 ∑ σ 𝑦 ∑ σ ∑ σ ∑ σ ∑ σ ∑ σ ∑ σ ∑ σ Input x Output y Ich I mag like Schokolade chocolate
  17. 17. © 2020 KNIME AG. All Rights Reserved. What are RNNs? 17 Image Source: Christopher Olah, https://colah.github.io/posts/2015-08-Understanding-LSTMs/
  18. 18. © 2020 KNIME AG. All Rights Reserved. From feed forward to recurrent neural networks 18 ∑ ∑ ∑ ∑ ∑ ∑ ∑ 𝒙 𝟐 𝒚 𝟑 𝒙 𝟐 𝒚 𝟑
  19. 19. © 2020 KNIME AG. All Rights Reserved. From feed forward to recurrent neural networks 19 ∑ ∑ ∑ 𝒙 𝟐 𝒚 𝟑 ∑ ∑ ∑ 𝒙 𝟐 𝒚 𝟑 ∑ ∑ ∑ 𝒙 𝟐 𝒚 𝟑 ∑ ∑ ∑ 𝒙 𝟐 𝒚 𝟑
  20. 20. © 2020 KNIME AG. All Rights Reserved. Simple RNN unit 20 Image Source: Christopher Olah, https://colah.github.io/posts/2015-08-Understanding-LSTMs/
  21. 21. © 2020 KNIME AG. All Rights Reserved. Limitations of simple layer structures 21 The “memory” of simple RNNs is sometimes too limited to be useful – “Cars drive on the ” (road) – “I love the beach – my favorite sound is the crashing of the “ (cars? glass? waves?)
  22. 22. © 2020 KNIME AG. All Rights Reserved. LSTM = Long Short Term Memory Unit • Special type of unit with three gates – Forget gate – Input gate – Output gate 22 Image Source: Christopher Olah, https://colah.github.io/posts/2015-08-Understanding-L
  23. 23. © 2020 KNIME AG. All Rights Reserved. Different network-structures and applications Many to Many 23 A A A <sos> A like sailing sailinglikeI I <eos> Language model Neural machine translation E E E like sailingI D D D Ich gehe gerne D segeln Ich gehe gerne
  24. 24. © 2020 KNIME AG. All Rights Reserved. Different network-structures and applications Many to one 24 A A A A I A like to go sailing English Language classification Text classification One to many A A A AA Couple onsailing a lake Image captioning
  25. 25. © 2020 KNIME AG. All Rights Reserved. How can we train a recurrent neural network for text generation • Decide between character, sub word, or word level – Word level • How we structure language • Length of sequences are much shorter – Character level • Smaller dictionary => much more manageable • Ability to generate new words – Sub-word level • Happy medium • What kind of data can we use? • How can we prepare our data for the training? 25
  26. 26. © 2020 KNIME AG. All Rights Reserved. 26 The Creative Side of AI: Naming New Products
  27. 27. © 2020 KNIME AG. All Rights Reserved. The Product Naming Phase Naming new products is not as easy as it might sound … 27 What happens when a new product is born?
  28. 28. © 2020 KNIME AG. All Rights Reserved. The Case Study: new line of outdoor clothing Outdoor or hiking clothing line needs new names that: • Evoke the feeling of nature • Sound familiar to customers • Stand out from the competition • Are not covered by copyright 28 • The team also needs many potential candidates to evaluate • Could a neural network help in this creative process?
  29. 29. © 2020 KNIME AG. All Rights Reserved. Let’s build a model to generate fictional mountain names! 29
  30. 30. © 2020 KNIME AG. All Rights Reserved. Quick sneak preview – Find the fake mountain names 30 Set 1 Barlock Mountain Casterland Mountain Shafford Peak Set 2 Roblin Hill Buxley Mountain Baldy Rock Set 3 Terrey Hill Walter Hill Little Buck Butte Correct answer: ALL!
  31. 31. © 2020 KNIME AG. All Rights Reserved. The Dataset 31 • 33,012 names of mountains in the US • Extracted from Wikipedia via a Wikidata query
  32. 32. © 2020 KNIME AG. All Rights Reserved. Many to many structure 32
  33. 33. © 2020 KNIME AG. All Rights Reserved. LSTM based Neural Network: Many to Many 33 0,1,0,0,0,0,0,0,0,...,0 0,0,0,1,0,0,0,0,0,...,0 Mount Baker u o M 0,0,0,0,0,0,1,0,0,...,0 0,0,0,0,0,0,0,1,0,...,0 0,0,0,0,1,0,0,0,0,...,0 t n 0,0,0,0,0,0,0,0,1,...,0 B 1,0,0,0,0,0,0,0,0,...,0 95=dictionarysize M samples 95=dictionarysize a b k m n z ... N=256 LSTM states M = max. Length of mountain name 0,0,0,1,0,0,0,0,0,...,0 u o 0,0,0,0,0,0,1,0,0,...,0 0,0,0,0,0,0,0,1,0,...,0 0,0,0,0,1,0,0,0,0,...,0 t n 0,0,0,0,0,0,0,0,1,...,0 B 1,0,0,0,0,0,0,0,0,...,0 M samples 0,1,0,0,0,0,0,0,0,...,0 a linear softmax N LSTMstates
  34. 34. © 2020 KNIME AG. All Rights Reserved. The KNIME Keras Integration 34 • Codeless GUI based • Fully opensource • Keras functions https://www.knime.com/deeplearning/keras
  35. 35. © 2020 KNIME AG. All Rights Reserved. Codeless? 35 LSTMstates
  36. 36. © 2020 KNIME AG. All Rights Reserved. Model training workflow • Read in and transform the input dataset • Define the structure of the neural network • Train the model using Keras Learner • Do a bit of post-training model editing to introduce temperature • Convert model to TensorFlow and save for use in deployment workflow 36
  37. 37. © 2020 KNIME AG. All Rights Reserved. Now that we have a model, how do we use it? 37
  38. 38. © 2020 KNIME AG. All Rights Reserved. Neural network: Code-free Layers of our neural network: • Input layer of size [?, 95] • LSTM for sequence analysis • Dropout layer to prevent overfitting • Dense layer 1 – linear activation • Dense layer 2 (output) – Softmax activation Once we’ve defined the network structure and pre-processed our input, we’re ready to train. 38
  39. 39. © 2020 KNIME AG. All Rights Reserved. Want to see the names? 39 Advantages: - They remind you of mountains. Do not they? - Sound familiar enough - Evoke the feeling of nature - No copyright issues - Automatic generation in 10 seconds - You can generate as many as you want - No people involved as of now Can this be considered a creative task?
  40. 40. © 2020 KNIME AG. All Rights Reserved. 40 Yo! AI Generated Rap Songs
  41. 41. © 2020 KNIME AG. All Rights Reserved. Bots & speaking styles 41 - Articulate - Chatty - Clean - Conversational - Crisp - Declamatory - Diffuse - Discoursive - Eloquent - Emphatic - Epigrammatic - Epistolary - Euphemistic - Flowery - Funny - Fluent - Formal - Gossipy - Idiomatic - Incoherent - Informal - Journalistic - Literary - Lyric - Ornate - Parenthetical - Pejorative - Picturesque - Poetic - Prolix - Punchy - Rambling - Rhetorical - Rough - Sesquipedalianhttps://writerswrite.co.za/60-words-used-to-describe-writing-or-speech-style/
  42. 42. © 2020 KNIME AG. All Rights Reserved. How should my bot speak? • I want my bot to answer in a polite tone in some situations, even almost poetic • I want my bot to answer in an affirmative tone in other situations, borderline impolite 42
  43. 43. © 2020 KNIME AG. All Rights Reserved. Let‘s start with impolite ... • Let‘s generate a rap song! • This is a free text generation problem • Similar to the problem of the product name generation, just on longer complete sentences • Let’s build a similar network from scratch! • Maybe not. • Let’s search Kathrin’s example on the KNIME Hub • and readapt it! 43
  44. 44. © 2020 KNIME AG. All Rights Reserved. On the KNIME Hub 44 https://hub.knime.com
  45. 45. © 2020 KNIME AG. All Rights Reserved. On the KNIME Hub 45 https://kni.me/w/mGO9nXhmjzIKiqHU
  46. 46. © 2020 KNIME AG. All Rights Reserved. LSTM based Neural Network 46 N 0,1,0,0,0,0,0,0,0,...,0 0,0,0,1,0,0,0,0,0,...,0 u o h 0,0,0,0,0,0,1,0,0,...,0 0,0,0,0,0,0,0,1,0,...,0 s 86=dictionarysize M past samples 86=dictionarysize a b e m n z ... Dictionary size = 86 256 LSTM states 100 past characters Training set = 23 popular rap songs LSTMstates
  47. 47. © 2020 KNIME AG. All Rights Reserved. Creative AI: the training workflow 47
  48. 48. © 2020 KNIME AG. All Rights Reserved. Creative AI: the deployment workflow 48
  49. 49. © 2020 KNIME AG. All Rights Reserved. don't need no advice You're not here and we both know why, so Move from me when you're extra Move from me with the passa I'm building up a house where they raised me You move with me I'll go crazy Don't switch on me, I got big plans We need to forward to the islands and get you gold, no spray tans I need you to stop running back to your ex he's a wasteman I wanna know how come we can never slash and stay friends I'm blem for real, I might just say how I feel Together forever Good morning, good afternoon, goodnight I'm here to talk about More Life On the Phostest, *** tike and push it And push it good for the cremm Mand a ray *** the but bo che turn me But dow of the Purb everydor get my mus, **** **** you move my home back hoot on the als to a lot on the crim just the yourh I'm a sixe and got a proup in the stake of the Ond I won't stop abusing it To groupie girls stop false accusing it Back to the music The Maybach roof is translucent **** got a problem Houston What up be The Let’s rappify some text Brick X6, Phey, cabe, make you feel soom the way (I smoke good!) I probably make (What?) More money in six months, Than what's in your papa's safe (I'm serious) Look like I robbed a bank (Okay Okay) I set it off like Queen Latifah 'Cause I'm living single I'm feeling cautious I ain't scream when they served a subpoena (Can't go back to jail) I heard that he a leader (Who pood, what to be *****' up The baugerout Black alro Black X6, Phantom White X6 looks like a panda Goin' out like I'm Montana Hundred killers, hundred hammers Black X6, Phantom White X6, panda Pockets swole, Danny Sellin' bar, candy Man I'm the macho like Randy The choppa go Oscar for Grammy **** **** pull up ya panty Hope you killas understand me Hey Panda, Panda Panda, Panda, Panda, Panda, Panda I got broads in Atlanta Twistin' dope, lean, and the Fanta Credit cards and the scammers Hittin' off licks in the bando 49 th eaf hard fortars I wanna lister for ald be gadg to call up in the Phantom Know *****, they come and kill you on the camera Big Rollie, it dancin' bigger than a Pandie Go Oscar for Grammy, **** pull up your panty Fill up I'ma flip it, I got **** pull up and peese vire Reepin me sher wing these hoes Don't settle for less one you lise one And my cas up in the Pun I arn’t **** with me If you wanted to These expensive, these is red bottoms These is bloody somes It the street to you heare it all while I be You in the club just to party I'm there, I get paid a fee I be in and out them banks so much I know they're tired of me Honestly, don't give a **** 'Bout who ain't fond of me Dropped two mixtapes in six months What **** working as hard as me? I don't bother with these hoes Don't let the truth to the young black youth But shorty's running with my ****, for the man, I as what I was gang affiliated, got on TV and told on me I guess that's why last winter she got so cold on me She s When the cat is away The mice shall play When the cat is away The mice shall play Never mind I'll find someone like you I wish nothing but the best For you too, don't forget me I beg, I lease, yua do not a aze me I around my dramy And when I'm finished, bring the yellow tape To tape off the scene of the slaughter Still gettin' swoll off bread and water I don't know if they *** or what Search a *** down, and grabbin' his *** And on the other hand, without a gun they can't get none But don't let it be a black and a white one 'Cause they'll slam ya down to the street top Black police showin' out for the white cop Ice Cube will swarm On any mother**** in a blue uniform Just cause I'm from, the CPT Punk police are afraid of man en I han to we have a keround hord **** it I'm right What more can I say to you? Get my grown man on Let's go! (What more can I say?) Now you know ass is willie When they got you in a mag For like half a billi And your ass ain't Lilly White That mean that shit you write must be illy Either that or your flow is silly It's both I don't mean to boast But damn if I don't brag Them crackers gonna act like I ain't on they ads The Martha Stewart That Yo! This post is about generating free text with a deep learning network particularly it is about This License refers to version of the GNU General Public License. Copyright also means copyright-bi
  50. 50. © 2020 KNIME AG. All Rights Reserved. 50 To be or not to be … Shakespearian text?
  51. 51. © 2020 KNIME AG. All Rights Reserved. Now for the poetic text ... • How would Shakespeare say it? • I can reuse the same LSTM based network • With a different training set • Training set is the full texts of – “King Lear” – “Othello” – “Much ado about nothing” • The only difference in the network is in the dictionary size 51 “Many a true word hath been spoken in jest.” ― William Shakespeare, King Lear “O, beware, my lord, of jealousy; It is the green-ey’d monster, which doth mock The meat it feeds on.” ― William Shakespeare, Othello “There was a star danced, and under that was I born.” ― William Shakespeare, Much Ado About Nothing
  52. 52. © 2020 KNIME AG. All Rights Reserved. Want to see some AI/Shakespeare text? To see you here before me. O my soul’s joy! I am a man a worm. – Gloucester I know not, sweet: I found your like a thief from the heart That pirchas will be well. The general speaks to be so Turn a man, I think, besoou. – Cassio I pray you, sir, to lie: in this hand is not a tend and talking of it; I would not be threaten dispatch. Our good old friend, Lay comforts the state, seek for him; I will grife you be faster’d! And the blessed course of dower: Net forth enough to do you; And that the Moor is defective in the letter! Abhorre, heaven, I will go sor; And the other too. – Othello I have a seet me such a trial of his speech, That he shall live the Moor in the lies That with daush’er Holds it is a most poor man, Whose welchers of the state, A man that many call of their life That have to lost the boy look to’t. – Regan Sir, to my sister? – Oswald I pray you, have your hand: you may receive it all to his sorrage, and makes the heavens Cassio lies that in the heart That I may speak: I’ll wast … 52 – Desdemona I pray, talk me of Cassio. Ay, so I hear the write to prive and there, That she would seen him that present so lich wored old wat, and the best conscionable than in this revolumance against him: There’s son against father, and thy father’s son and the best of our time hath no exalse your counsel watch The worst is not a tender to the warlike isle, That sunded must needs take the shame which the revenges of the self-same malk than the best of our times; keeps our fortunes fend for bearing to a strength, Sight in their natures, letting go safely by the rack: I swear ‘tis, to be so That she will send back my messenger. – Gloucester I see the rust in the stocks. – King Lear What a trifore be some cartiou, I can tell my way Than should be assurather, despise my brother; That I have passed him, tell me I was every think of fear, That she may be honest yet he hath confess’d in him entertains and think the next way to stain it, That the main commet in the least Would fail her breath, That she may … -bick, Remade me any thing to his sword To his salt and most hidden loose to be so for sings, but not in a libutt of his matter than that shall be sure as will be soldye As master compary, do not live in traitor. Bless thy five wits! -Kent O pity! Sir, where is the patience now, That this is so far from the sea and some bidings to dismantle So many folds of save and honest. -Brabantio I must not think the Turk of Cassio in the strange metting the cribles of a charmer be the reviling of libe to say That I can deceive him to the best advantage, In her prophetic fairs of a little to presently at your powers; whereof I thank you, sir. -Albany Gloucester, I will prove upancy of his sport and first accuriors and guard and talking on the white. -King Lear Where are the thief? Thou shalt never have the captains at the letter To the Moor and thing we have not the better shall be sure as worth if he be anger— -Regan I pray you, have a countend more than think to do a proclaim’d there of my heart, Hot and the best of our that she could else was not a toman. Good faith, hold, I beseech your grace,— -King Lear Then there’s so much to him. -King Lear Thou hast seen a part of this plainness I had thought to see the ride on the sea sevel never second to take the foul fiend. Still through the way that shoulds: I know not what. -Othello What dost thou see her? O unhappy girl! Sir, this desperate, bastard! what news? Mistage on my father! -King Lear What say’st thou st? -Kent Sir, I do know you; And to the Moor and rain, I beserve her take my sisters? -King Lear No, no, no, no! where’t the castle. Enter Gloucester, with King of France and Cordelia Goneril Sir, I had thought it not. -Iago Indeed! -Othello Indeed! ay, indeed: desceme, sir. -Cornwall Sighing, would! -Othello What dost thou see how this paper shall I carry out a fellow there, that makes his son a poor poor power That makes his son When the rain came to the devil wrath! I have lost the king from the cold wind: Take them what the … – Othello O my fair warrior! – Desdemona My dear Othello! – Othello It gives me wonder great as my content SCENE I. Venice. A street. Enter Roderigo and Iago This License refers to version of the GNU General Public License. Copyright also means copyright- Smokin’ on cookie in the hotbox cookie F*****’ on your b**** she a thot, thot, thot thot Cookin fellow,
  53. 53. © 2020 KNIME AG. All Rights Reserved. 53 Conclusions
  54. 54. © 2020 KNIME AG. All Rights Reserved. Can AI be creative? Yo! This post is about generating free text with a deep learning network particularly it is about Brick X6, Phey, cabe, make you feel soom the way (I smoke good!) I probably make (What?) More money in six months, Than what's in your papa's safe (I'm serious) Look like I robbed a bank (Okay Okay) I set it off like Queen Latifah 'Cause I'm living single I'm feeling cautious I ain't scream when they served a subpoena (Can't go back to jail) I heard that he a leader (Who pood, what to be f*****' up The baugerout Black alro Black X6, Phantom White X6 looks like a panda Goin' out like I'm Montana Hundred killers, hundred hammers Black X6, Phantom White X6, panda Pockets swole, Danny Sellin' bar, candy Man I'm the macho like Randy The choppa go Oscar for Grammy B**** n**** pull up ya panty Hope you killas understand me Hey Panda, Panda Panda, Panda, Panda, Panda, Panda I got broads in Atlanta Twistin' dope, lean, and the Fanta Credit cards and the scammers Hittin' off licks in the bando 54 This License refers to version of the GNU General Public License. Copyright also means copyright-bick, Remade me any thing to his sword To his salt and most hidden loose to be so for sings, but not in a libutt of his matter than that shall be sure as will be soldye As master compary, do not live in traitor. Bless thy five wits! -Kent O pity! Sir, where is the patience now, That this is so far from the sea and some bidings to dismantle So many folds of save and honest. -Brabantio I must not think the Turk of Cassio in the strange metting the cribles of a charmer be the reviling of libe to say That I can deceive him to the best advantage, In her prophetic fairs of a little to presently at your powers; whereof I thank you, sir. -Albany Gloucester, I will prove upancy of his sport and first accuriors and guard and talking on the white. -King Lear Where are the thief? Thou shalt never have the captains at the letter To the Moor and thing we have not the better shall be sure as worth if he be anger— -Regan I pray you, have a countend more than think to do a proclaim’d there of my heart, Hot The words save, honest, thief, master, traitor, and deceive seem to fit the context. Notice also that the dialogue sprouting from the start text of the license agreement interestingly involves mainly minor, less tragic characters from the plays. Caro amico ti scrivo così mi distraggo un po'E siccome sei molto lontano più forte ti scriverò. Da quella prima folla strana, che aveva preso il suo nome, e di correre alla casa di don Abbondio, con un viso bene di non poterci andar la casa del padre Cristoforo, e gli disse che s'avvicinava all'uscio, e si mise a sparse di corsa, e di stare a sé, verso la strada di servizio, chiesto le parole che gli andavan dall'altra stanza, e con la sua condizione de' cappuccini, e di consigli ricerche di confidenza delle gride, nel suo passaggio, se non pensava con una certa ripugnanza a casa sua, che andavano a scomparire in un campo di buone ragioni che avevan potuto raccogliere i suoi pensieri, e di sopra non senza interrogare, che la sua avventura aveva fatto predicare, e con la forza d'un fatto come fuggitive che aveva preso il suo nome, e di correre alla casa di don Abbondio, con un cappuccino di quella sorte, con un certo sospiro, alzando le sue finestre, e le diede un'occhiata in carrozza. Si vendano a metter nelle mani di chi era stato a sedere sur una strada così fatta con le braccia in
  55. 55. © 2020 KNIME AG. All Rights Reserved. References • https://www.datanami.com/2019/04/04/product-naming-with-deep-learning/ • https://opendatascience.com/how-to-use-deep-learning-to-write-shakespeare/ • https://customerthink.com/ai-generated-rap-songs/ 55
  56. 56. © 2020 KNIME AG. All Rights Reserved. Free Book as a Thank You Free Copy of e-book: “Practicing Data Science. A Collection of Case Studies” from KNIME Press https://www.knime.com/knimepress with this code: AUSTIN-0120 56
  57. 57. © 2019 KNIME AG. All Rights Reserved. The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by KNIME AG under license from KNIME GmbH, and are registered in the United States. KNIME® is also registered in Germany. The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by KNIME AG under license from KNIME GmbH, and are registered in the United States. KNIME® is also registered in Germany. Thank you! Questions?
  58. 58. © 2020 KNIME AG. All Right Reserved. Transfer Learning Corey Weisinger
  59. 59. © 2020 KNIME AG. All Rights Reserved. What is Transfer Learning? • Transfer learning can be defined as the attempt to utilize predictive ability in one input / output space to aid in the learning of new input spaces, output spaces, or both. • In this use case the input space, 3x64x64, remains the same, but the output space changes from 1000 image categories to 3 cancer types. A B CNew
  60. 60. © 2020 KNIME AG. All Rights Reserved. Transfer Learning – Basic Example Input Data Sepal Length Sepal Width Petal Length Petal Width Output Prediction Safe to Eat Lots of data likely needed to tune weights for logistic regression Input Data Sepal Length Sepal Width Petal Length Petal Width Output Prediction Safe to Eat Transferred Learning Type of Flower Easy to train small decision tree on little data StandardLearningTransferLearning
  61. 61. © 2020 KNIME AG. All Rights Reserved. Cancer Cell Classification Use Case • Uses Keras Deep Learning model. • Learning Transferred from VGG16 image classifier • Completely code free with KNIME’s Deep Learning Integration • Can be adapted to a wide range of image classification problems This Photo by Unknown Author is licensed under CC BY-SA VGG16 Cat VGG16 New CLL Original Task New Task Image From: https://ome.grc.nia.nih.g ov/iicbu2008/lymphoma/ index.html
  62. 62. © 2020 KNIME AG. All Rights Reserved. Why is it helpful? • Deep Learning requires tons of data • Image classification requires tons and tons of data • We have ~5,000 labeled cancer images • VGG16 was trained on more than 1,000,000 images from ImageNet dataset. 5k 1,200k CANCER CELLS IMAGENET DATA POINTS
  63. 63. © 2020 KNIME AG. All Rights Reserved. Feature Extraction in Neural Networks • One of the benefits of deep learning is its ability to perform its own feature engineering • This can occur through vector embedding, convolutional layers, LSTM layers etc. Figure available under Creative Commons Attribution 4.0 International
  64. 64. © 2020 KNIME AG. All Rights Reserved. Convolutional Layer • Instead of connecting every neuron to the new layer a sliding window is used • Use when your data has spatial relationships – 2D: Image – 3D: Video • Some convolutions may detect edges or corners, while others may detect cats, dogs, or street signs inside an image Image from: https://towardsdatascience.com/a-comprehensive-guide- to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53
  65. 65. © 2020 KNIME AG. All Rights Reserved. Convolutional Layer Cont. • Stride: – Distance between adjacent windows (windows can overlap) • Window/Kernel/Filter size: – dimension(s) of windows • Pooling: – When the convolution maps many to one. Commonly taking the max or average value in a window. • Benefits: – Fewer weights than dense layers – Less prone to overfitting – Feature extraction useful for transfer learning Image from: https://www.freecodecamp.org/news/an-intuitive-guide- to-convolutional-neural-networks-260c2de0a050/
  66. 66. © 2020 KNIME AG. All Rights Reserved. VGG16 Model • Several blocks of small convolution layers followed by max pooling layers • Trained on ImageNet dataset – Over 1,000,000 images from 1,000 thousand classes • Won ILSVRC-2013 competition in single network category Final Layer We used VGG16 Architecture: https://neurohive.io/en/popular-networks/vgg16/
  67. 67. © 2020 KNIME AG. All Rights Reserved. Everything in KNIME 1. Download Data Set 2. Pre-Process Data Set 1. Crop, Normalize, Rearrange 3. Configure Network 1. Load VGG16 2. Add and Freeze Layers 4. Train Network 5. Post-Process Predictions 1. Group images, create Classification Workflow on KNIME Hub: https://kni.me/w/1_w_dxiltpHmsBQA
  68. 68. © 2020 KNIME AG. All Rights Reserved. Pre-Processing • Input size – VGG16: 3x64x64 image – Us: 1388x1040x3 image • Image Values – VGG16: Normalized 0-1 – Cancer Cells: 0-256 • Pre-Processing – Multiple patches per image – Normalize (per image) – Re-arrange dimensions Image processing nodes part of KNIME Image Processing Extension: https://kni.me/e/Uq6QE1IQIqG4q_mp Create Patches Metanode
  69. 69. © 2020 KNIME AG. All Rights Reserved. Configuring the Network • Load VGG16 Network • Add custom layers to produce 3 class predictions • Freeze original VGG16 layers – Keep feature detection intact – Significantly reduces number of parameters to be learned Keras nodes part of KNIME Deep Learning – Keras Integration: https://kni.me/e/XOee1uZPrzE36EPH # create the base pre-trained model model = VGG16(weights='imagenet', include_top=False, input_shape=(64, 64, 3)) # add Flatten, dense, and dropout layers model.add(Flatten()) model.add(Dense(32), activation=‘ReLu’) model.add(Dropout(0.5)) # add a fully-connected layer output layer model.add(Dense(3, activation=‘Softmax’)) # first: train only the top layers for layer in model.layers[0:-4]: layer.trainable = False
  70. 70. © 2020 KNIME AG. All Rights Reserved. Training the Network • We train the newly configured network for 25 epochs • Only updating the weights in our unfrozen end layers • Using Categorical Cross Entropy Loss function Learning Monitor, part of Keras Network Learner node
  71. 71. © 2020 KNIME AG. All Rights Reserved. Post-Processing • Combine image patches – Average prediction by class – Use max probability as classification • Scoring – High Accuracy for small training set – High Cohen’s Kappa value shows significant divergence from random chance
  72. 72. © 2020 KNIME AG. All Rights Reserved. Finding the Example
  73. 73. © 2020 KNIME AG. All Rights Reserved. Related KNIME Blog Articles • Original use case, involves some coding: – https://www.knime.com/blog/u sing-the-new-knime-deep- learning-keras-integration-to- predict-cancer-type-from- histopatholog • Updated use case, totally code free: – https://www.knime.com/blog/tr ansfer-learning-made-easy-with- deep-learning-keras-integration
  74. 74. © 2020 KNIME AG. All Rights Reserved. Citations • Convolution Layer: https://towardsdatascience.com/a- comprehensive-guide-to-convolutional-neural-networks-the- eli5-way-3bd2b1164a53 • Cancer Cell Slide: https://ome.grc.nia.nih.gov/iicbu2008/lymphoma/index.html • VGG16 Architecture: https://neurohive.io/en/popular- networks/vgg16/
  75. 75. © 2020 KNIME AG. All Rights Reserved. KNIME Course Books Course books downloadable from KNIME Press https://www.knime.com/knimepress with code: AUSTIN-0120
  76. 76. © 2019 KNIME AG. All Rights Reserved. The KNIME® trademark and logo and OPEN FOR INNOVATION® trademark are used by KNIME AG under license from KNIME GmbH, and are registered in the United States. KNIME® is also registered in Germany. Thank you! Questions?
  77. 77. © 2020 KNIME AG. All Rights Reserved. Next Meetup • Planning for April, then afterward June • Next location: IBM campus on Burnet Road • April Meetup topic ideas? – Active Learning talk – KNIME Use Cases talk – Raw Data to Deployment Learnathon – Time Series Learnathon – ...or something else? • Want to present on KNIME and your project? Let us know!
  • SouravDuttMSBlackBel

    Jun. 13, 2021

An introduction to recurrent neural networks and LSTM units followed by some example applications for language modeling.

Views

Total views

272

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

44

Shares

0

Comments

0

Likes

1

×