SlideShare ist ein Scribd-Unternehmen logo
1 von 81
Downloaden Sie, um offline zu lesen
Knowledge-based
Music Recommendation
Models, Algorithm and Exploratory Search
Michel BUFFA Reviewer
Mounia LALMAS Reviewer
Gaël RICHARD Examiner
Tommaso DI NOIA Examiner
Pietro MICHIARDI Examiner
Benoit HUET Thesis Director
Raphäel TRONCY Thesis Co-Director
ThesisCommittee
PhD Candidate
Pasquale Lisena
11 October 2019
1. Music
in particular Classical Music
2. Knowledge
Graphs as part of Semantic Web
technologies
3. ML
techniques
applied to Music KG
in particular for recommendation
What’s my
thesis about
Why Classical Music?
4
CLASSICALPOPULAR VS
5
M. Lasar (2011). Digging into Pandora’s Music Genome with musicologist Nolan Gasser.
https://arstechnica.com/tech-policy/2011/01/digging-into-pandoras-music-genome-with-musicologist-nolan-gasser/
When it comes to classical
music, on the other hand, it's
much more about the
composition itself, because
even though the interpretation
can vary in various subtle ways.
CLASSICALPOPULAR VS
For pop music the experience of
the music is really defined by
the recording.
6
CLASSICALPOPULAR VS
Track-based Work-based
70 years of history
Thousand years
from Gregorian chant to a work written last
Tuesday
Songs Multi-movement works
Major, minor
Polyphonic, homophonic,
monophonic
7
M. Schedl (2015) Towards Personalizing Classical Music Recommendations. 2015 IEEE International Conference
on Data Mining Workshop (ICDMW), Atlantic City, NJ, USA, pp. 1366-1367. https://doi.org/10.1109/ICDMW.2015.8
“Fans of classical music
are underrepresented
on social media and
music streaming
platforms.”
● Less data
● Less detailed metadata
● Less involved in research
Music recommendation
research
Classical music
recommendation
research
8
Data Metadata
Data which describes other data
composer
composition date
genre
performer
key
derivation type
1801
9
Title, opus, movement
Who is the composer?
Who is the performer?
online music approach
Track as “atomic unit”
10
music archives approach
Work as “aggregation unit”
● genre
● date
● author
● title(s)
● …
● publication
● performance
● recordings
● books
● ...
11
Which model best represents these
rich data for final users and music
scholars?
What strategies to adopt for building
a music Knowledge Graph?
How to make these data accessible
to researchers and developers?
How can graph-based algorithms
support music recommender systems?
What information can be extracted
from editorial playlists?
Is Graph representation also suitable
for music content?
Research Questions
RQ1
RQ2
RQ3
RQ4
RQ5
RQ6
Roadmap
A. Music Model & Vocabularies
B. Data Conversion
C. Web APIs for KG
Building a
Music graph
Exploiting the
Music knowledge
PART I PART II
12
A. Embeddings and Similarity
B. Playlists and Weights
C. Learning MIDI Embeddings
13
Improve music description to foster
music exchange and reuse
Travel to the heart of the musical
archives in France’s greatest
institutions
Connect sources, multiply usage,
enrich user experience
14
pic: https://flic.kr/p/29YHAqY
Building a
Music graph
PART I
15
What is a Knowledge Graph?
source: https://hackernoon.com/wtf-is-a-knowledge-graph-a16603a1a25f
It is a specific kind of knowledge base
which is:
● a directed graph
connections between nodes are first-class citizens
● semantic
the meaning of the connections are part of the data itself
● smart
allows graph-computing techniques and algorithms
● alive
easy to extend, access, reuse
Semantic Web technologies realize graphs in
which nodes and properties linking them are
identified by URIs.
16
17
pic:https://pxhere.com/it/photo/1523259
I.A Music Model &
Vocabularies
Building a
Music graph
Which model
to represent
this richness?
musicologists
libraries
musical museums
conservatories
radios
concert halls
RQ1
18
- One of the first example of describing music
using Semantic Web
- Extend FRBR, Timeline Ontology, Event
Ontology
- Uses vocabularies for Keys, Musical
Instrument (by MusicBrainz), Genres (DBpedia)
Y. Raimond, S. Abdallah, M. Sandler, and F. Giasson (2007). The Music Ontology.
In 15th International Conference on Music Information Retrieval (ISMIR). 417–422
State of the art
The Music Ontology
Building a Music graph Music Model & Vocabularies I.A
19
The DOREMUS Model
- Relies on Linked Data and Semantic
Web principles
○ everything is a URI
○ RDF model
- Music specific extension of FRBRoo
- Event-based pattern: the knowledge is
represented in modules (triangles) which
describe events that give birth to
work/expression
FRBR
museum
information
bibliographic
records
P. Choffé and F. Leresche (2016). DOREMUS: connecting sources, enriching catalogues
and user experience. In 24th IFLA World Library and Information Congress.
Building a Music graph Music Model & Vocabularies I.A
20
F14
Work
F22
Expression
M2
Opus
Statement
F28
Expression
Creation
R3 is
realized in
E7
Activity
5
1
“Sonate pour violoncelle et piano no 1”@fr
“Sonates" , "Sonata in F"
Ludwig van
Beethoven
Ludwig von Beethoven
composer
compositeur@fr
compositore@it
R17 created
R19createda
realizationof
U17 has opus
statement
U12 has
genre
P102 has title
U31 had
function of
type
P14 carried
out by
P9
consists of
P4 has time
span1796
Sonata
sonata@it , sonate@fr ,
klaviersonate@de
M42 Performed
Expression
Creation
M43
Performed
Expression
Berlin
P4 has time
span
1796
P7 took
place at
F24 Publication
Expression
F30
Publication
Event
P4 has time
span
1797
P7 took
place at
Vienna
U4 had princeps
publication
U54 is performed
expression of
P165
incorporates
1770
1827
P98
born
P100
died
U11 has key
F Major
F Dur@de , Fa majeur@fr,
Fa maggiore@it , Fa mayor@es
M6
Casting
M23
Casting
Detail
U13
has
casting
1
U30
quantity
U2
foresees
mop
Piano
Pianoforte@it
Fortepian@pl
M23
Casting
Detail
1
U30
quantity
U2
foresees
mop
Cello
Violoncello@it
Violoncelle@fr
F15
Complex
Work
F19
Publication
Work
M44
Performed
Work
U5 had
premiere
U38 has
descriptive
expression
R10 has member
21
“Sax”@en
“Saxophone”@en
“Saxofone”@pt
“Sassofono”@it
“Saxophone”@fr
Alternate
labels
Alternate
languages
“English term is preferred globally”
Notes
“Woodwinds”@en
“Legni”@it
Hierarchy
“Baritone Saxophone”@en
Example: http://data.doremus.org/vocabulary/iaml/mop/wsa
Controlled Vocabularies
Building a Music graph Music Model & Vocabularies I.A
GENRES
Diabolo
IAML
Itema3
Redomi
RAMEAU
Medium of performance
MIMO
Itema3
IAML
Diabolo
RAMEAU
Redomi
Musical keys
Modes
Catalogues
Derivation types
Functions
more available at
http://data.doremus.org/vocabularies
23 families of vocabularies · 11,000+ concepts · 610 links between terms
INTERLINKED
INTERLINKED
P. Lisena et al. (2018). Controlled Vocabularies for Music Metadata.
In 19th International Conference on Music Information Retrieval (ISMIR). Paris, France.
Controlled Vocabularies
Building a Music graph Music Model & Vocabularies I.A
These and additional
competency questions have
been collected by experts from
our partner institutions and used
as requirements and validation
for the model.
https://github.com/DOREMUS-ANR/knowledge-base/tree/master/
query-examples
23
P. Lisena et al. (2017) Modeling the Complexity of Music Metadata in
Semantic Graphs for Exploration and Discovery. In (ISMIR’17) 4th
International Workshop on Digital Libraries for Musicology (DLfM’17),
Shanghai, China.
Building a Music graph Music Model & Vocabularies
Which works have been
composed
by Mozart when he was
<10?
How many works have
been composed and
performed for the 1st
time in the same city?
Which composers had the
chance to direct their own
work in a performance
during the last decade?
I.A
24
Which chamber music works have been
composed in the 19th century by
Scandinavian composers?
Edvard Grieg
1843 - 1907
Work
Genre
>1800
AND
<=1900
CHAMBER
MUSIC
Composition date
?composed
by
nationality
part of
SCANDINAVIA
Building a Music graph Music Model & Vocabularies I.A
I.B Data Conversion
Building a
Music graph
26
Music archives have
very detailed knowledge
PROBLEMS
● Multiple formats
○ sometimes complex parsing is required
● No possible interoperability
● Need for discovering overlapping
knowledge
● Information codified as free text
○ different practices in codifying the same
information (“Op. 27 n. 2” - “Op. 27 no 2”)
○ wrong fields, typos, wrong punctuation
● Not always publicly accessible
pic: wikimedia commons
Building a Music graph Data Conversion I.B
Ryszard Kruk S. andl McDaniel B. (2009). Goals of Semantic Digital Libraries.
Source datasets
27
Works
62 550 | XML
Scores
9 154 | XML
Concerts
340 609 | XML
Discs
9 500 | XML
Works
6 846 | UNIMARC
Scores
30 319 | UNIMARC
Concerts
5 164 | XML
Discs
8 602 | XML
Works
135 940 | INTERMARC
Scores
89 184 | INTERMARC
(3 different XML sources)
Building a Music graph Data Conversion I.B
28
001 FRBNF139081882FR
100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827
144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur
001 FRBNF139081882FR
100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827
144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur
LANG TITLE MOP OPUS KEY
MARC FILE
Building a Music graph Data Conversion I.B
29
001 FRBNF139081882FR
100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827
144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur
MARC FILE
NUM SUB
Building a Music graph Data Conversion I.B
marc2rdf
MARC PARSER
FREE TEXT
INTERPRETER
STRING 2
VOCABULARY
MARC
files
vocabularies
1st performance in Moscow, December 29, 1956,
by Mstislav Rostropovich on cello and A. Dedukhin on piano
“ ”
mapping
rules
Building a Music graph Data Conversion I.B
RDF
graph
What strategies to adopt for building a music Knowledge Graph?RQ2
31
INTERMARC
marc2rdf
UNIMARC
EUTERPE
XML
ITEMA3
XML
euterpe
converter
itema3
converter
GRAPH
BNF
GRAPH
PHILHARMONIE GRAPH EUTERPE GRAPH ITEMA3
diabolo
converter
DIABOLO
XML
GRAPH DIABOLO
STRING 2 VOCABULARY
Building a Music graph Data Conversion I.B
32
What is in the Knowledge Graph?
89.872
persons
(composers,
performers, …)
18.075
corporate bodies
(orchestras, chorus,
publishers, …)
357.451
musical
works
16k components
4k derived works
193.412
concerts and
studio recordings
469.131
performed work
3.833
foreseen
concerts
31.296
publications
48.006
scores
Building a Music graph Data Conversion I.B
33
pic: https://www.flickr.com/photos/franganillo/2643351571
I.C Web APIs for KG
Building a
Music graph
Pëtr Il'ič Čajkovskij
Pyotr Ilyich Tchaikovsky
Пётр Ильич Чайковский
GALLERY OF COMPOSERS
Antonio Vivaldi
Ludwig van Beethoven Johann Sebastian Bach
Jean Sébastien Bach [FR]
34
SELECT * WHERE {
?composer a foaf:Person ;
foaf:name ?name ;
foaf:depiction ?img .
}
34
Pëtr Il'ič Čajkovskij
Pyotr Ilyich Tchaikovsky
Пётр Ильич Чайковский
GALLERY OF COMPOSERS
Antonio Vivaldi
Ludwig van Beethoven Johann Sebastian Bach
Jean Sébastien Bach [FR]
Building a Music graph Web APIs for KG I.C
35
-- W3C specification
SPARQL result
JSON format
"bindings": [{
"composer": { "type": "uri",
"value": "http://data.doremus.org/artist/0b9d963c-bfd7-337d-b6c3-c874f5e62125"
},
"name": { "type": "literal",
"value": "Petr Ilʹič Čajkovskij"
},
"img": { "type": "uri",
"value": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg"
}
}, {
"composer": { "type": "uri",
"value": "http://data.doremus.org/artist/0b9d963c-bfd7-337d-b6c3-c874f5e62125"
},
"name": { "type": "literal",
"value": "Piotr Ilitch Tchaikovski"
},
"img": { "type": "uri",
"value": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg"
}
}, {
"composer": { "type": "uri",
"value": "http://data.doremus.org/artist/b34f92ab-ad86-361b-a8b8-5c3a4db784d0"
},
"name": { "type": "literal",
"value": "Antonio Vivaldi"
},
"img": { "type": "uri",
"value": "http://.../Antonio_Vivaldi.jpg"
}
}, ...
SAME
DIFFERENT
SAME
DIFFERENT
How to make these data
accessible to researchers and
developers?
RQ3
36
[{
"id": "http://data.doremus.org/artist/0b9d963c...",
"name": [
"Petr Ilʹič Čajkovskij"
"Piotr Ilitch Tchaikovski"
],
"image": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg"
},
{
"id": "http://data.doremus.org/artist/b34f92ab...",
"name": "Antonio Vivaldi",
"image": "http://.../Antonio_Vivaldi.jpg"
}]
2
names
1
picture
Building a Music graph Web APIs for KG I.C
37
skip irrelevant
metadata
reducing and
parsing
merging “rows”
mapping to
different structures
Building a Music graph Web APIs for KG I.C
Booth et al. (2019) Toward Easier RDF. In W3C Workshop on
Web Standardization for Graph Data.
38
SPARQL Transformer
/D2KLab/sparql-transformer
● JS and Python library
● A JSON-based syntax
○ template + query
● Integration in grlc.io for
web api development
{
"proto": {
"id" : "?composer",
"name": "$foaf:name$required",
"image": "$foaf:depiction$required"
},
"$where": [
"?composer a ecrm:E21_Person"
],
"$limit": 100
}
Building a Music graph Web APIs for KG I.C
Lisena P. et al. (2019). Easy Web API Development
with SPARQL Transformer. In ISWC’19.
39
SPARQL Transformer
Building a Music graph Web APIs for KG I.C
QUERIES*
n. objects
(original)
n. objects
(transformed)
1.Born_in_Berlin 1132 573
2.German_musicians 290 257
3.Musicians_born_in_Berlin 172 109
4.Soccer_players 78 70
5.Games 1020 981
Evaluation #1: Queries’ results
* from https://wiki.dbpedia.org/onlineaccess
Evaluation #2: User Survey
55 subjects
Used in
Overhead < 0.1 seconds
40
pic: https://www.flickr.com/photos/ncculture/2065959239
Exploiting the
Music knowledge
PART II
41
discover new
music
improve their streaming
music experience
background for
their activities
FINAL USERS MUSIC EXPERTS
playlist producing
help for concert
programming
automatic radio
broadcasting
How can graph-based algorithms support
music recommender systems?
RQ4
42
Antonio Vivaldi
Autumn. I Allegro
Tomaso Albinoni
Symphony n. 3 NEXT
SEED
TARGET
How to find it?
43
S. Oramas, V. C. Ostuni, T. Di Noia, X. Serra, and E. Di Sciascio. Sound and music recommendation with knowledge graphs.
ACM Trans. Intell. Syst. Technol. 8, 2, Article 21 (October 2016), 21 pages.
http://dx.doi.org/10.1145/2926718
The vector representation of the
item i is computed on his
neighborhood of length l.
More two items share
entities/property at a certain
distance, more those items can be
considered similar.
State of the art
Item neighborhood mapping
44
https://musiclynx.github.io/#/artist/ad79836d-9849-44df-8789-180bbc823f3c/Antonio%2520Vivaldi
Alo Allik, Florian Thalmann, and Mark Sandler. 2018. MusicLynx: Exploring Music Through Artist Similarity Graphs.
In The Web Conference 2018. Demo Track, pp 167-170.
https://doi.org/10.1145/3184558.3186970
● Access to different knowledge sources
● Maximum Degree Weighted (MDW): links to very
large categories (i.e. Living People) are discouraged
with respect to more significant ones.
State of the art
MusicLynx
II.A Embeddings
and Similarity
pic https://pxhere.com/es/photo/1168914
Exploiting the
Music knowledge
46
Word Embeddings (e.g. word2vec)
corpus of document -> vectors that represent the semantic
distribution of words in the text
Graph Embeddings (e.g. node2vec)
set of random walks -> vectors that represent the semantic distribution
of entity in the graph
Exploiting the Music knowledge Embeddings and Similarity II.A
Main idea: nodes that occurs in similar contexts
(neighborhood of nodes in a graph) are more similar, and
will be closer in the vector space.
Aditya Grover and Jure Leskovec. node2vec: Scalable Feature Learning for Networks.
In 22nd ACM SIGKDD , 2016.
46
Some problems:
● Our dataset was constantly growing
● The amount of nodes is huge
● Different purposes for recommendation:
○ radio broadcasting
○ concert programming
○ final users
47
computational-wise and time-wise
expensive
(multiple run of node2vec
on huge amount of data)
Exploiting the Music knowledge Embeddings and Similarity II.A
48
Compute embeddings at
simple features level
period of time
musical key
medium of performance
genre
...
Exploiting the Music knowledge Embeddings and Similarity II.A
Solution
49
vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl
Example: MoP
The clarinet is more similar to the oboe or to the cello?
vocabulary:iaml/mop/svc
Exploiting the Music knowledge Embeddings and Similarity II.A
50
vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl
vocabulary:iaml/mop/w vocabulary:iaml/mop/s
vocabulary:iaml/mop/svc
Example: MoP
Exploiting the Music knowledge Embeddings and Similarity II.A
Graph of vocabularies
51
vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl
vocabulary:iaml/mop/svc
Casting Detail Casting Detail
Casting Detail
Casting
862 times
Casting
1213 times
Example: MoP
Exploiting the Music knowledge Embeddings and Similarity II.A
Graph of usage
52
SPARQL
endpoint
subgraph
(edgelist)
selection of
interesting
properties
(i.e. skos:broader)
vectors
embedding
NODE2VEC
s 1.34 0.98 0.20
w 1.44 1.21 0.31
svc 0.14 1.31 1.48
wcl -1.2 1.90 0.85
wob -0.83 2.32 1.03
Pasquale Lisena et al. Controlled Vocabularies for Music Metadata.
19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, September 2018.
Exploiting the Music knowledge Embeddings and Similarity II.A
53
vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl
The clarinet is more similar to the oboe or to the cello?
vocabulary:iaml/mop/svc
0.506 0.562
Exploiting the Music knowledge Embeddings and Similarity II.A
Example: MoP
54
VECTOR SPACE OF MoPs
ethnic
chordophones
ethnic flutes
percussions
brass
orchestra
woodwinds
orchestra
strings
rare strings
Exploiting the Music knowledge Embeddings and Similarity II.A
55
VECTOR SPACE OF GENRES
Exploiting the Music knowledge Embeddings and Similarity II.A
56
Combine embeddings at
complex features level
artists
works
playlists
Exploiting the Music knowledge Embeddings and Similarity II.A
Example: Artists
Exploiting the Music knowledge Embeddings and Similarity II.A
58
MOP
embeddings:
MOPGENRE KEY
Artist’s features:
BIRTH
DATE
DEATH
DATE
CASTING
WORKS
GENRE
WORKS
KEYS
WORKS
PLAYED
MOP
-0.02 0.01 0.01 0.00 -0.01 -0.02 0.01 0.00 -0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -0.00 0.00 0.00 0.07 -0.03 0.07 -0.02 -0.01 0.19 0.02 0.69 -0.19 -0.14 0.08 0.03 0.03 0.00 0.08 null null null null null -0.06 0.07 0.02 -0.03 0.00
Artist vector
BIRTH
PLACE
DEATH
PLACE
FUNCTION
FUNCTGeoNamesGeoNamesTime
DIMENSIONALITY REDUCTION (PCA)
Time
AVG AVG AVG AVG AVG
Exploiting the Music knowledge Embeddings and Similarity II.A
Some data are unknown or not applicable
null null null null null
59P. Lisena, R. Troncy (2017). Combining music specific embeddings for computing artist similarity.
In 18th International Society for Music Information Retrieval Conference (ISMIR), Late-breaking & demo track.
percentage of missing
dimensions in artist 2
with respect to artist 1
Exploiting the Music knowledge Embeddings and Similarity II.A
Example: Artists
60
Do all the properties have the same
importance?
Exploiting the Music knowledge Embeddings and Similarity II.A
II.B Playlists and
Weights
pic:https://pixabay.com/photo-791076/
Exploiting the
Music knowledge
Which
information is
possible to
extract from
editorial
playlists?
RQ5
Idea: there are “unaware”
rules that experts apply
when realising a playlist.
62
● Use playlists to give a weight to the influence of each dimension.
● No GOLD STANDARD available, creation of a “silver” one
Radio France Playlists
(50)
Spotify Playlists
(65)
ITEMA3 Concerts (624)
Philharmonie Concerts (186)
● Radio France Web Radio (7 channels)
● Realised by experts
● Classical section of Spotify app
● Realised by Spotify staff
● Real concerts that took place in Paris
(studio + concert hall)
Exploiting the Music knowledge Playlists and Weights II.B
63
variance within < variance between
HOMOGENEOUS
good for recommendation
variance within > variance between
INHOMOGENEOUS
bad for recommendation
Exploiting the Music knowledge Playlists and Weights II.B
F test statistic = variance between / variance within Weights
evaluation
64
Exploiting the Music knowledge Playlists and Weights II.B
● 7 music experts
from partner institutions
● given the seed:
○ put bad items
in the trash
○ sort according
to preference
evaluation
65
Exploiting the Music knowledge Playlists and Weights II.B
The study of variance help us to identify which dimensions
should be promoted for better recommendations.
66
Exploiting the Music knowledge Playlists and Weights II.B
Under experimentation in
live.philharmoniedeparis.fr
The role of titles:
Title2Rec
Exploiting the
Music knowledge
“Relax Driving”
Johannes Brahms
Symphony n.3
“Beach Party”
Luis Fonsi
Despacito
68
Title2Rec: training
Exploiting the Music knowledge Playlists and Weights II.B
Content
(id of tracks)
Playlists
SEQUENTIAL
EMBEDDINGS
(word2vec)
CLUSTERS
of playlists
Titles
DOCUMENTS
fastText
MODEL
69
yy :) christmas litmas guardians christmas christmas holiday christmas christmas the good stuff. xmas himym christmas pop xmas
country happy holidays holidays christmas christmas hits 25 just cause stay christmas tis the season 🎄 christmas 🎄 christmas
oldbutgold christmas christmas vibes christmas strong christmas winter wonderland christmas time december 15 xmas christmas
christmas pop flight christmas deep christmas vibes christmas oldies work in progress christmas christmas playlist christmas music
josh 🎄 christmas blah christmas & chill depression secret christmas christmas & chill christmas love :) christmas elite :) christmas
special songs christmas christmas christmas jams jessica its lit classy pump up graduation at the moment .... christmas christmas
christmas music good old days christmas mix christmas music 80s rock christmas 2015 xmas christmas christmas christmas
christmas vibes 2017 songs christmas vibes!! christmas music holidays christmas 2016! christmas christmas club music summer
2015 christmasssss christmassss christmas christmas christmas christmas!! christmas christmas feels christmas christmas(::
christmas playlist great christmas playlist christmas & chill christmas christmas trap blast from the past christmas 2016 classics grad
christmas christmas christmas christmas yessss christmas christmas rihanna christmas christmas songs christmas 2016!!!!! good
vibes christmas christmas songs christmas christmas christmas favorites christmas christmas 2016 🎄 christmas last christmas
christmas all my friends christmas christmas !! chirstmas the weeknd christmas 2015 christmas christmas lyrical party music wake up
happy vibes 🎄 christmas calm country winter christmas christmas christmas pop christmas af ❄ christmas feel good :)) christmas
christmas af christmas jams moana christmas merry christmas! christmas playlist christmas christmas silly love songs christmas </3
school 🎄 christmas christmas music christmas christmas music 🎄 christmas x-mas christmas bops christmas beachin' dance jamz
christmas new wave its christmas christmas 🎄 christmas indie 2 christmas 1980 christmas jams christmas 2015 sunrise christmas
christmas playlist christmas jams christmas white ella chirstmas sleep :))))) christmas random christmas dance christmas christmas
december; christmas christmas favs christmas old christmas songs ~holidaze~ christmas christmas music xmas christmas holidays
december christmas christmas christmas baby wedding music tis the season christmas relax holidays!! 🎅 🏼 christmas christmas
christmas december '15 christmas!! christmas new songs christmas christmas
Exploiting the Music knowledge Playlists and Weights II.B
Title2Rec: training
70
Title2Rec: predicting
Exploiting the Music knowledge Playlists and Weights II.B
Given a new title:
● found the most similar titles
among the known ones
● propose the most popular
tracks among those titles
Evaluated on Spotify’s
Million Playlists Datasets
in the context of the
RecSys Challenge 2018
in the challenge:
#37 over 112
#13 over 31
II.C Learning
MIDI Embeddings
Exploiting the
Music knowledge
Is Graph
representation
also suitable for
music content?
RQ5
72
MIDI2vec
Apply graph technologies to MIDI
● Transform MIDI flow in a graph
● Apply node2vec for learning graph
embeddings
Exploiting the Music knowledge Learning MIDI Embeddings II.C
MIDI
Group
of
Notes
Pitch
Duration
Program
Time
Signature
Tempo
Velocity
+
+
+
+
+
73
Experiment: genre and metadata prediction
Dataset 1: SLAC
250 MIDI, balanced on 5/10 genres
Accuracies on cross-fold validation:
Dataset 2: MuseData
438 MIDI, unbalanced, linked to DOREMUS
Accuracies on cross-fold validation:
Exploiting the Music knowledge Learning MIDI Embeddings II.C
Baseline: McKay et al (2010)
74
Exploratory Search Engine
overture.doremus.org
Chatbot
chatbot.doremus.org
Emotion Detection
data.doremus.org/emotion
75
Which model best represents these rich data for final
users and music scholars?
DOREMUS model and Vocabularies
What strategies to adopt for building a music
Knowledge Graph?
marc2rdf and other converters
result: the DOREMUS Knowledge Graph
How to make these data accessible to researchers
and developers?
SPARQL Transformer reshapes and
merges the results for easy use
RQ1
RQ2
RQ3
Main contributions
76
How can graph-based algorithms support music
recommender systems?
Embedding approach with generation
and recombination of partial vectors
Which information is possible to extract from
editorial playlists?
A study of editorial playlists, for
weighting a recommender system
Title2Rec: recommend music by the
title of the playlist
Graph representation is suitable also for music
content?
MIDI2vec: learning MIDI graph
embeddings
RQ4
RQ5
RQ6
Main contributions
77
Future Work (1/2)
Short Term
● Studies on simplifications of the ontology (schema.org)
● Domain-based NLP for text-field information extraction
Long Term
● Strategies for modeling librarian information
representing meta-information on a 2nd level (RDF*)
Modeling and accessing a KG
78
Future Work (2/2)
Short term
● Split the dataset in historical period
more precise training, faster performances
● Title2Rec + similarity-based recommender system
application for editors
● Experiment MIDI embeddings on larger dataset
Long term
● Gold standard dataset of classical music playlists
● Combining our strategy with more traditional ones (CF)
● MIDI ontology: extend and use in MIDI2vec
Knowledge-aware Recommender system
Publications
Conference Poster&Demo Journal Tutorial Workshop
EKAW'16
ISWC'16
EKAW'16 2016
ISWC'17 X2
ISMIR'17
K-CAP'17 DLfM'17 2017
ISWC'18 X2
ISMIR'18
ISMIR'18 BIBLIOTHEK -
Forschung
und Praxis
ESWC'18 RecSys'18
TheWebConf'18
2018
ISWC'19 2019
PC Member
ISWC’18 P&D, SAAM’18, DLfM’18, ISWC’19 P&D, K-CAP’19
as sub-reviewer: KAARS’18, TheWebConf’19
Student
Supervision
2 Master Thesis supervisions
10 Semester Projects supervisions
Lecturer for WebInt and Aalto BootCamp
Talks
Des Catalogues au Web des Données
- BnF, Paris
Classical Music and Knowledge Graphs
- Semantic Web course, PoliTo
- WAI meeting, VU Amsterdam
- Research seminar, Deezer, Paris
80
References (1/2)
● M. Schedl (2015) Towards Personalizing Classical Music Recommendations. 2015 IEEE International Conference
on Data Mining Workshop (ICDMW), Atlantic City, NJ, USA, pp. 1366-1367.
● Y. Raimond, S. Abdallah, M. Sandler, and F. Giasson (2007). The Music Ontology. In 15th International
Conference on Music Information Retrieval (ISMIR). 417–422
● P. Choffé and F. Leresche (2016). DOREMUS: connecting sources, enriching catalogues and user experience.
In 24th IFLA World Library and Information Congress.
● P. Lisena et al. (2018). Controlled Vocabularies for Music Metadata. In 19th International Conference on Music
Information Retrieval (ISMIR). Paris, France.
● P. Lisena et al. (2017) Modeling the Complexity of Music Metadata in Semantic Graphs for Exploration and
Discovery. In 4th International Workshop on Digital Libraries for Musicology (DLfM’17), Shanghai, China.
● Booth et al. (2019) Toward Easier RDF. In W3C Workshop on Web Standardization for Graph Data.
● Lisena P. et al. (2019). Easy Web API Development with SPARQL Transformer. In ISWC’19.
● S. Oramas, V. C. Ostuni, T. Di Noia, X. Serra, and E. Di Sciascio. Sound and music recommendation with
knowledge graphs. ACM Trans. Intell. Syst. Technol. 8, 2, Article 21 (October 2016), 21 pages.
81
References (2/2)
● Alo Allik, Florian Thalmann, and Mark Sandler (2018). MusicLynx: Exploring Music Through Artist Similarity
Graphs. In The Web Conference 2018. Demo Track, pp 167-170.
● Aditya Grover and Jure Leskovec. (2016) node2vec: Scalable Feature Learning for Networks. In 22nd ACM
SIGKDD.
● McKay, C., Burgoyne, J., Hockman, J., B. L. Smith, J.,Vigliensoni, G., and Fujinaga, I. (2010). Evaluating the
Genre Classification Performance of Lyrical Features Relative to Audio, Symbolic and Cultural
Features. In ISMIR 2011, Utrecht, The Netherlands
● Meroño-Peñuela, A., Hoekstra, R., Gangemi, A., Bloem, P., de Valk, R., Stringer, B., Janssen, B., de Boer, V.,Allik,
A., Schlobach, S., et al. (2017). The MIDI Linked Data Cloud. In ISWC 2017, Vienna, Austria.
● Huang, A. and Wu, R. (2016). Deep Learning for Music. Computing Research Repository (CoRR),
https://arxiv.org/abs/1606.04930 .
● Peter Knees and Markus Schedl (2013). A survey of music similarity and recommendation from music
context data. ACM Trans. Multimedia Comput. Commun. Appl. 10, 1, Article 2 (December 2013), 21 pages.
● Palumbo, Rizzo, Troncy. (2017) entity2rec: Learning user-item relatedness from knowledge graphs for
top-N item recommendation. In RECSYS 2017, Como, Italy.

Weitere ähnliche Inhalte

Ähnlich wie Knowledge-based Music Recommendation

Gym And Fitness Gymnasium
Gym And Fitness GymnasiumGym And Fitness Gymnasium
Gym And Fitness GymnasiumTamika Brown
 
Music in Movement – an interactive guide to contemporary classical music (DRA...
Music in Movement – an interactive guide to contemporary classical music (DRA...Music in Movement – an interactive guide to contemporary classical music (DRA...
Music in Movement – an interactive guide to contemporary classical music (DRA...FIAT/IFTA
 
Introduction to Fast by Professor Mark Sandler
Introduction to Fast by  Professor Mark SandlerIntroduction to Fast by  Professor Mark Sandler
Introduction to Fast by Professor Mark SandlerFASTIMPACT
 
Finbar m usic industry vision 2014
Finbar m usic industry vision 2014Finbar m usic industry vision 2014
Finbar m usic industry vision 2014Finbar O'Hanlon
 
Difesa del suolo e pianificazione territoriale
Difesa del suolo e pianificazione territorialeDifesa del suolo e pianificazione territoriale
Difesa del suolo e pianificazione territorialepibinko
 
Enhancing a Digital Sheet Music Collection A report for LIS-435 ...
 Enhancing a Digital Sheet Music Collection A report for LIS-435 ... Enhancing a Digital Sheet Music Collection A report for LIS-435 ...
Enhancing a Digital Sheet Music Collection A report for LIS-435 ...crysatal16
 
The Negative Impact Of Audio Recording On Tinfoil Processes
The Negative Impact Of Audio Recording On Tinfoil ProcessesThe Negative Impact Of Audio Recording On Tinfoil Processes
The Negative Impact Of Audio Recording On Tinfoil ProcessesAlyssa Jones
 
Analysis Synthesis Comparison
Analysis Synthesis ComparisonAnalysis Synthesis Comparison
Analysis Synthesis ComparisonJim Webb
 
The Listening Experience Database
The Listening Experience DatabaseThe Listening Experience Database
The Listening Experience DatabaseAlessandro Adamou
 
Final Doccument (Finished Copy)
Final Doccument (Finished Copy)Final Doccument (Finished Copy)
Final Doccument (Finished Copy)Edmund Hull
 
Future Producers
Future ProducersFuture Producers
Future ProducersNick Cicero
 
Stop Looking and Start Listening
Stop Looking and Start ListeningStop Looking and Start Listening
Stop Looking and Start ListeningBecky Stewart
 
Information about the Modern Music industry
Information about the Modern Music industry Information about the Modern Music industry
Information about the Modern Music industry bobonthedrums
 
Defining New Languages For Querying Rdf Data
Defining New Languages For Querying Rdf DataDefining New Languages For Querying Rdf Data
Defining New Languages For Querying Rdf DataAmber Rodriguez
 
O.Ogunyemi MANUSCRIPT2016
O.Ogunyemi MANUSCRIPT2016O.Ogunyemi MANUSCRIPT2016
O.Ogunyemi MANUSCRIPT2016Titus Ogunyemi
 
Digital Scholarship Intersection
Digital Scholarship IntersectionDigital Scholarship Intersection
Digital Scholarship IntersectionDavid De Roure
 

Ähnlich wie Knowledge-based Music Recommendation (20)

Gym And Fitness Gymnasium
Gym And Fitness GymnasiumGym And Fitness Gymnasium
Gym And Fitness Gymnasium
 
Music in Movement – an interactive guide to contemporary classical music (DRA...
Music in Movement – an interactive guide to contemporary classical music (DRA...Music in Movement – an interactive guide to contemporary classical music (DRA...
Music in Movement – an interactive guide to contemporary classical music (DRA...
 
Iaml montréal 2012
Iaml montréal 2012Iaml montréal 2012
Iaml montréal 2012
 
Introduction to Fast by Professor Mark Sandler
Introduction to Fast by  Professor Mark SandlerIntroduction to Fast by  Professor Mark Sandler
Introduction to Fast by Professor Mark Sandler
 
Finbar m usic industry vision 2014
Finbar m usic industry vision 2014Finbar m usic industry vision 2014
Finbar m usic industry vision 2014
 
Difesa del suolo e pianificazione territoriale
Difesa del suolo e pianificazione territorialeDifesa del suolo e pianificazione territoriale
Difesa del suolo e pianificazione territoriale
 
Enhancing a Digital Sheet Music Collection A report for LIS-435 ...
 Enhancing a Digital Sheet Music Collection A report for LIS-435 ... Enhancing a Digital Sheet Music Collection A report for LIS-435 ...
Enhancing a Digital Sheet Music Collection A report for LIS-435 ...
 
The Negative Impact Of Audio Recording On Tinfoil Processes
The Negative Impact Of Audio Recording On Tinfoil ProcessesThe Negative Impact Of Audio Recording On Tinfoil Processes
The Negative Impact Of Audio Recording On Tinfoil Processes
 
Analysis Synthesis Comparison
Analysis Synthesis ComparisonAnalysis Synthesis Comparison
Analysis Synthesis Comparison
 
The Listening Experience Database
The Listening Experience DatabaseThe Listening Experience Database
The Listening Experience Database
 
Final Doccument (Finished Copy)
Final Doccument (Finished Copy)Final Doccument (Finished Copy)
Final Doccument (Finished Copy)
 
Future Producers
Future ProducersFuture Producers
Future Producers
 
The MIDI Linked Data Cloud
The MIDI Linked Data CloudThe MIDI Linked Data Cloud
The MIDI Linked Data Cloud
 
Stop Looking and Start Listening
Stop Looking and Start ListeningStop Looking and Start Listening
Stop Looking and Start Listening
 
Kraftwerk Influence
Kraftwerk InfluenceKraftwerk Influence
Kraftwerk Influence
 
Information about the Modern Music industry
Information about the Modern Music industry Information about the Modern Music industry
Information about the Modern Music industry
 
Music
MusicMusic
Music
 
Defining New Languages For Querying Rdf Data
Defining New Languages For Querying Rdf DataDefining New Languages For Querying Rdf Data
Defining New Languages For Querying Rdf Data
 
O.Ogunyemi MANUSCRIPT2016
O.Ogunyemi MANUSCRIPT2016O.Ogunyemi MANUSCRIPT2016
O.Ogunyemi MANUSCRIPT2016
 
Digital Scholarship Intersection
Digital Scholarship IntersectionDigital Scholarship Intersection
Digital Scholarship Intersection
 

Kürzlich hochgeladen

biosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsbiosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsSafaFallah
 
Pests of ragi_Identification, Binomics_Dr.UPR
Pests of ragi_Identification, Binomics_Dr.UPRPests of ragi_Identification, Binomics_Dr.UPR
Pests of ragi_Identification, Binomics_Dr.UPRPirithiRaju
 
Main Exam Applied biochemistry final year
Main Exam Applied biochemistry final yearMain Exam Applied biochemistry final year
Main Exam Applied biochemistry final yearmarwaahmad357
 
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...Sérgio Sacani
 
geometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsgeometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsHassan Jolany
 
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptxQ3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptxArdeniel
 
Principles & Formulation of Hair Care Products
Principles & Formulation of Hair Care  ProductsPrinciples & Formulation of Hair Care  Products
Principles & Formulation of Hair Care Productspurwaborkar@gmail.com
 
Basic Concepts in Pharmacology in molecular .pptx
Basic Concepts in Pharmacology in molecular  .pptxBasic Concepts in Pharmacology in molecular  .pptx
Basic Concepts in Pharmacology in molecular .pptxVijayaKumarR28
 
Application of Foraminiferal Ecology- Rahul.pptx
Application of Foraminiferal Ecology- Rahul.pptxApplication of Foraminiferal Ecology- Rahul.pptx
Application of Foraminiferal Ecology- Rahul.pptxRahulVishwakarma71547
 
MARSILEA notes in detail for II year Botany.ppt
MARSILEA  notes in detail for II year Botany.pptMARSILEA  notes in detail for II year Botany.ppt
MARSILEA notes in detail for II year Botany.pptaigil2
 
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfSUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfsantiagojoderickdoma
 
Applied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxApplied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxmarwaahmad357
 
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPirithiRaju
 
Role of herbs in hair care Amla and heena.pptx
Role of herbs in hair care  Amla and  heena.pptxRole of herbs in hair care  Amla and  heena.pptx
Role of herbs in hair care Amla and heena.pptxVaishnaviAware
 
Alternative system of medicine herbal drug technology syllabus
Alternative system of medicine herbal drug technology syllabusAlternative system of medicine herbal drug technology syllabus
Alternative system of medicine herbal drug technology syllabusPradnya Wadekar
 
Lehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptLehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptSachin Teotia
 
RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024suelcarter1
 
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...marwaahmad357
 
CW marking grid Analytical BS - M Ahmad.docx
CW  marking grid Analytical BS - M Ahmad.docxCW  marking grid Analytical BS - M Ahmad.docx
CW marking grid Analytical BS - M Ahmad.docxmarwaahmad357
 

Kürzlich hochgeladen (20)

biosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibioticsbiosynthesis of the cell wall and antibiotics
biosynthesis of the cell wall and antibiotics
 
Pests of ragi_Identification, Binomics_Dr.UPR
Pests of ragi_Identification, Binomics_Dr.UPRPests of ragi_Identification, Binomics_Dr.UPR
Pests of ragi_Identification, Binomics_Dr.UPR
 
Main Exam Applied biochemistry final year
Main Exam Applied biochemistry final yearMain Exam Applied biochemistry final year
Main Exam Applied biochemistry final year
 
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
Digitized Continuous Magnetic Recordings for the August/September 1859 Storms...
 
geometric quantization on coadjoint orbits
geometric quantization on coadjoint orbitsgeometric quantization on coadjoint orbits
geometric quantization on coadjoint orbits
 
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptxQ3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
Q3W4part1-SSSSSSSSSSSSSSSSSSSSSSSSCI.pptx
 
Principles & Formulation of Hair Care Products
Principles & Formulation of Hair Care  ProductsPrinciples & Formulation of Hair Care  Products
Principles & Formulation of Hair Care Products
 
Basic Concepts in Pharmacology in molecular .pptx
Basic Concepts in Pharmacology in molecular  .pptxBasic Concepts in Pharmacology in molecular  .pptx
Basic Concepts in Pharmacology in molecular .pptx
 
Application of Foraminiferal Ecology- Rahul.pptx
Application of Foraminiferal Ecology- Rahul.pptxApplication of Foraminiferal Ecology- Rahul.pptx
Application of Foraminiferal Ecology- Rahul.pptx
 
MARSILEA notes in detail for II year Botany.ppt
MARSILEA  notes in detail for II year Botany.pptMARSILEA  notes in detail for II year Botany.ppt
MARSILEA notes in detail for II year Botany.ppt
 
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdfSUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
SUKDANAN DIAGNOSTIC TEST IN PHYSICAL SCIENCE ANSWER KEYY.pdf
 
Applied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docxApplied Biochemistry feedback_M Ahwad 2023.docx
Applied Biochemistry feedback_M Ahwad 2023.docx
 
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdfPests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
Pests of wheat_Identification, Bionomics, Damage symptoms, IPM_Dr.UPR.pdf
 
Role of herbs in hair care Amla and heena.pptx
Role of herbs in hair care  Amla and  heena.pptxRole of herbs in hair care  Amla and  heena.pptx
Role of herbs in hair care Amla and heena.pptx
 
Cheminformatics tools and chemistry data underpinning mass spectrometry analy...
Cheminformatics tools and chemistry data underpinning mass spectrometry analy...Cheminformatics tools and chemistry data underpinning mass spectrometry analy...
Cheminformatics tools and chemistry data underpinning mass spectrometry analy...
 
Alternative system of medicine herbal drug technology syllabus
Alternative system of medicine herbal drug technology syllabusAlternative system of medicine herbal drug technology syllabus
Alternative system of medicine herbal drug technology syllabus
 
Lehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.pptLehninger_Chapter 17_Fatty acid Oxid.ppt
Lehninger_Chapter 17_Fatty acid Oxid.ppt
 
RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024RCPE terms and cycles scenarios as of March 2024
RCPE terms and cycles scenarios as of March 2024
 
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
001 Case Study - Submission Point_c1051231_attempt_2023-11-23-14-08-42_ABS CW...
 
CW marking grid Analytical BS - M Ahmad.docx
CW  marking grid Analytical BS - M Ahmad.docxCW  marking grid Analytical BS - M Ahmad.docx
CW marking grid Analytical BS - M Ahmad.docx
 

Knowledge-based Music Recommendation

  • 1. Knowledge-based Music Recommendation Models, Algorithm and Exploratory Search Michel BUFFA Reviewer Mounia LALMAS Reviewer Gaël RICHARD Examiner Tommaso DI NOIA Examiner Pietro MICHIARDI Examiner Benoit HUET Thesis Director Raphäel TRONCY Thesis Co-Director ThesisCommittee PhD Candidate Pasquale Lisena 11 October 2019
  • 2. 1. Music in particular Classical Music 2. Knowledge Graphs as part of Semantic Web technologies 3. ML techniques applied to Music KG in particular for recommendation What’s my thesis about
  • 5. 5 M. Lasar (2011). Digging into Pandora’s Music Genome with musicologist Nolan Gasser. https://arstechnica.com/tech-policy/2011/01/digging-into-pandoras-music-genome-with-musicologist-nolan-gasser/ When it comes to classical music, on the other hand, it's much more about the composition itself, because even though the interpretation can vary in various subtle ways. CLASSICALPOPULAR VS For pop music the experience of the music is really defined by the recording.
  • 6. 6 CLASSICALPOPULAR VS Track-based Work-based 70 years of history Thousand years from Gregorian chant to a work written last Tuesday Songs Multi-movement works Major, minor Polyphonic, homophonic, monophonic
  • 7. 7 M. Schedl (2015) Towards Personalizing Classical Music Recommendations. 2015 IEEE International Conference on Data Mining Workshop (ICDMW), Atlantic City, NJ, USA, pp. 1366-1367. https://doi.org/10.1109/ICDMW.2015.8 “Fans of classical music are underrepresented on social media and music streaming platforms.” ● Less data ● Less detailed metadata ● Less involved in research Music recommendation research Classical music recommendation research
  • 8. 8 Data Metadata Data which describes other data composer composition date genre performer key derivation type 1801
  • 9. 9 Title, opus, movement Who is the composer? Who is the performer? online music approach Track as “atomic unit”
  • 10. 10 music archives approach Work as “aggregation unit” ● genre ● date ● author ● title(s) ● … ● publication ● performance ● recordings ● books ● ...
  • 11. 11 Which model best represents these rich data for final users and music scholars? What strategies to adopt for building a music Knowledge Graph? How to make these data accessible to researchers and developers? How can graph-based algorithms support music recommender systems? What information can be extracted from editorial playlists? Is Graph representation also suitable for music content? Research Questions RQ1 RQ2 RQ3 RQ4 RQ5 RQ6
  • 12. Roadmap A. Music Model & Vocabularies B. Data Conversion C. Web APIs for KG Building a Music graph Exploiting the Music knowledge PART I PART II 12 A. Embeddings and Similarity B. Playlists and Weights C. Learning MIDI Embeddings
  • 13. 13 Improve music description to foster music exchange and reuse Travel to the heart of the musical archives in France’s greatest institutions Connect sources, multiply usage, enrich user experience
  • 15. 15 What is a Knowledge Graph? source: https://hackernoon.com/wtf-is-a-knowledge-graph-a16603a1a25f It is a specific kind of knowledge base which is: ● a directed graph connections between nodes are first-class citizens ● semantic the meaning of the connections are part of the data itself ● smart allows graph-computing techniques and algorithms ● alive easy to extend, access, reuse Semantic Web technologies realize graphs in which nodes and properties linking them are identified by URIs.
  • 16. 16
  • 17. 17 pic:https://pxhere.com/it/photo/1523259 I.A Music Model & Vocabularies Building a Music graph Which model to represent this richness? musicologists libraries musical museums conservatories radios concert halls RQ1
  • 18. 18 - One of the first example of describing music using Semantic Web - Extend FRBR, Timeline Ontology, Event Ontology - Uses vocabularies for Keys, Musical Instrument (by MusicBrainz), Genres (DBpedia) Y. Raimond, S. Abdallah, M. Sandler, and F. Giasson (2007). The Music Ontology. In 15th International Conference on Music Information Retrieval (ISMIR). 417–422 State of the art The Music Ontology Building a Music graph Music Model & Vocabularies I.A
  • 19. 19 The DOREMUS Model - Relies on Linked Data and Semantic Web principles ○ everything is a URI ○ RDF model - Music specific extension of FRBRoo - Event-based pattern: the knowledge is represented in modules (triangles) which describe events that give birth to work/expression FRBR museum information bibliographic records P. Choffé and F. Leresche (2016). DOREMUS: connecting sources, enriching catalogues and user experience. In 24th IFLA World Library and Information Congress. Building a Music graph Music Model & Vocabularies I.A
  • 20. 20 F14 Work F22 Expression M2 Opus Statement F28 Expression Creation R3 is realized in E7 Activity 5 1 “Sonate pour violoncelle et piano no 1”@fr “Sonates" , "Sonata in F" Ludwig van Beethoven Ludwig von Beethoven composer compositeur@fr compositore@it R17 created R19createda realizationof U17 has opus statement U12 has genre P102 has title U31 had function of type P14 carried out by P9 consists of P4 has time span1796 Sonata sonata@it , sonate@fr , klaviersonate@de M42 Performed Expression Creation M43 Performed Expression Berlin P4 has time span 1796 P7 took place at F24 Publication Expression F30 Publication Event P4 has time span 1797 P7 took place at Vienna U4 had princeps publication U54 is performed expression of P165 incorporates 1770 1827 P98 born P100 died U11 has key F Major F Dur@de , Fa majeur@fr, Fa maggiore@it , Fa mayor@es M6 Casting M23 Casting Detail U13 has casting 1 U30 quantity U2 foresees mop Piano Pianoforte@it Fortepian@pl M23 Casting Detail 1 U30 quantity U2 foresees mop Cello Violoncello@it Violoncelle@fr F15 Complex Work F19 Publication Work M44 Performed Work U5 had premiere U38 has descriptive expression R10 has member
  • 21. 21 “Sax”@en “Saxophone”@en “Saxofone”@pt “Sassofono”@it “Saxophone”@fr Alternate labels Alternate languages “English term is preferred globally” Notes “Woodwinds”@en “Legni”@it Hierarchy “Baritone Saxophone”@en Example: http://data.doremus.org/vocabulary/iaml/mop/wsa Controlled Vocabularies Building a Music graph Music Model & Vocabularies I.A
  • 22. GENRES Diabolo IAML Itema3 Redomi RAMEAU Medium of performance MIMO Itema3 IAML Diabolo RAMEAU Redomi Musical keys Modes Catalogues Derivation types Functions more available at http://data.doremus.org/vocabularies 23 families of vocabularies · 11,000+ concepts · 610 links between terms INTERLINKED INTERLINKED P. Lisena et al. (2018). Controlled Vocabularies for Music Metadata. In 19th International Conference on Music Information Retrieval (ISMIR). Paris, France. Controlled Vocabularies Building a Music graph Music Model & Vocabularies I.A
  • 23. These and additional competency questions have been collected by experts from our partner institutions and used as requirements and validation for the model. https://github.com/DOREMUS-ANR/knowledge-base/tree/master/ query-examples 23 P. Lisena et al. (2017) Modeling the Complexity of Music Metadata in Semantic Graphs for Exploration and Discovery. In (ISMIR’17) 4th International Workshop on Digital Libraries for Musicology (DLfM’17), Shanghai, China. Building a Music graph Music Model & Vocabularies Which works have been composed by Mozart when he was <10? How many works have been composed and performed for the 1st time in the same city? Which composers had the chance to direct their own work in a performance during the last decade? I.A
  • 24. 24 Which chamber music works have been composed in the 19th century by Scandinavian composers? Edvard Grieg 1843 - 1907 Work Genre >1800 AND <=1900 CHAMBER MUSIC Composition date ?composed by nationality part of SCANDINAVIA Building a Music graph Music Model & Vocabularies I.A
  • 26. 26 Music archives have very detailed knowledge PROBLEMS ● Multiple formats ○ sometimes complex parsing is required ● No possible interoperability ● Need for discovering overlapping knowledge ● Information codified as free text ○ different practices in codifying the same information (“Op. 27 n. 2” - “Op. 27 no 2”) ○ wrong fields, typos, wrong punctuation ● Not always publicly accessible pic: wikimedia commons Building a Music graph Data Conversion I.B Ryszard Kruk S. andl McDaniel B. (2009). Goals of Semantic Digital Libraries.
  • 27. Source datasets 27 Works 62 550 | XML Scores 9 154 | XML Concerts 340 609 | XML Discs 9 500 | XML Works 6 846 | UNIMARC Scores 30 319 | UNIMARC Concerts 5 164 | XML Discs 8 602 | XML Works 135 940 | INTERMARC Scores 89 184 | INTERMARC (3 different XML sources) Building a Music graph Data Conversion I.B
  • 28. 28 001 FRBNF139081882FR 100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827 144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur 001 FRBNF139081882FR 100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827 144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur LANG TITLE MOP OPUS KEY MARC FILE Building a Music graph Data Conversion I.B
  • 29. 29 001 FRBNF139081882FR 100 $313891295$w.0..b.....$aBeethoven$mLudwig van$d1770-1827 144 $w....b.fre.$aSonates$bPiano$pOp. 27, no 2$tDo dièse mineur MARC FILE NUM SUB Building a Music graph Data Conversion I.B
  • 30. marc2rdf MARC PARSER FREE TEXT INTERPRETER STRING 2 VOCABULARY MARC files vocabularies 1st performance in Moscow, December 29, 1956, by Mstislav Rostropovich on cello and A. Dedukhin on piano “ ” mapping rules Building a Music graph Data Conversion I.B RDF graph What strategies to adopt for building a music Knowledge Graph?RQ2
  • 31. 31 INTERMARC marc2rdf UNIMARC EUTERPE XML ITEMA3 XML euterpe converter itema3 converter GRAPH BNF GRAPH PHILHARMONIE GRAPH EUTERPE GRAPH ITEMA3 diabolo converter DIABOLO XML GRAPH DIABOLO STRING 2 VOCABULARY Building a Music graph Data Conversion I.B
  • 32. 32 What is in the Knowledge Graph? 89.872 persons (composers, performers, …) 18.075 corporate bodies (orchestras, chorus, publishers, …) 357.451 musical works 16k components 4k derived works 193.412 concerts and studio recordings 469.131 performed work 3.833 foreseen concerts 31.296 publications 48.006 scores Building a Music graph Data Conversion I.B
  • 33. 33 pic: https://www.flickr.com/photos/franganillo/2643351571 I.C Web APIs for KG Building a Music graph Pëtr Il'ič Čajkovskij Pyotr Ilyich Tchaikovsky Пётр Ильич Чайковский GALLERY OF COMPOSERS Antonio Vivaldi Ludwig van Beethoven Johann Sebastian Bach Jean Sébastien Bach [FR]
  • 34. 34 SELECT * WHERE { ?composer a foaf:Person ; foaf:name ?name ; foaf:depiction ?img . } 34 Pëtr Il'ič Čajkovskij Pyotr Ilyich Tchaikovsky Пётр Ильич Чайковский GALLERY OF COMPOSERS Antonio Vivaldi Ludwig van Beethoven Johann Sebastian Bach Jean Sébastien Bach [FR] Building a Music graph Web APIs for KG I.C
  • 35. 35 -- W3C specification SPARQL result JSON format "bindings": [{ "composer": { "type": "uri", "value": "http://data.doremus.org/artist/0b9d963c-bfd7-337d-b6c3-c874f5e62125" }, "name": { "type": "literal", "value": "Petr Ilʹič Čajkovskij" }, "img": { "type": "uri", "value": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg" } }, { "composer": { "type": "uri", "value": "http://data.doremus.org/artist/0b9d963c-bfd7-337d-b6c3-c874f5e62125" }, "name": { "type": "literal", "value": "Piotr Ilitch Tchaikovski" }, "img": { "type": "uri", "value": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg" } }, { "composer": { "type": "uri", "value": "http://data.doremus.org/artist/b34f92ab-ad86-361b-a8b8-5c3a4db784d0" }, "name": { "type": "literal", "value": "Antonio Vivaldi" }, "img": { "type": "uri", "value": "http://.../Antonio_Vivaldi.jpg" } }, ... SAME DIFFERENT SAME DIFFERENT How to make these data accessible to researchers and developers? RQ3
  • 36. 36 [{ "id": "http://data.doremus.org/artist/0b9d963c...", "name": [ "Petr Ilʹič Čajkovskij" "Piotr Ilitch Tchaikovski" ], "image": "http://.../Pyotr_Ilyich_Tchaikovsky.jpg" }, { "id": "http://data.doremus.org/artist/b34f92ab...", "name": "Antonio Vivaldi", "image": "http://.../Antonio_Vivaldi.jpg" }] 2 names 1 picture Building a Music graph Web APIs for KG I.C
  • 37. 37 skip irrelevant metadata reducing and parsing merging “rows” mapping to different structures Building a Music graph Web APIs for KG I.C Booth et al. (2019) Toward Easier RDF. In W3C Workshop on Web Standardization for Graph Data.
  • 38. 38 SPARQL Transformer /D2KLab/sparql-transformer ● JS and Python library ● A JSON-based syntax ○ template + query ● Integration in grlc.io for web api development { "proto": { "id" : "?composer", "name": "$foaf:name$required", "image": "$foaf:depiction$required" }, "$where": [ "?composer a ecrm:E21_Person" ], "$limit": 100 } Building a Music graph Web APIs for KG I.C Lisena P. et al. (2019). Easy Web API Development with SPARQL Transformer. In ISWC’19.
  • 39. 39 SPARQL Transformer Building a Music graph Web APIs for KG I.C QUERIES* n. objects (original) n. objects (transformed) 1.Born_in_Berlin 1132 573 2.German_musicians 290 257 3.Musicians_born_in_Berlin 172 109 4.Soccer_players 78 70 5.Games 1020 981 Evaluation #1: Queries’ results * from https://wiki.dbpedia.org/onlineaccess Evaluation #2: User Survey 55 subjects Used in Overhead < 0.1 seconds
  • 41. 41 discover new music improve their streaming music experience background for their activities FINAL USERS MUSIC EXPERTS playlist producing help for concert programming automatic radio broadcasting How can graph-based algorithms support music recommender systems? RQ4
  • 42. 42 Antonio Vivaldi Autumn. I Allegro Tomaso Albinoni Symphony n. 3 NEXT SEED TARGET How to find it?
  • 43. 43 S. Oramas, V. C. Ostuni, T. Di Noia, X. Serra, and E. Di Sciascio. Sound and music recommendation with knowledge graphs. ACM Trans. Intell. Syst. Technol. 8, 2, Article 21 (October 2016), 21 pages. http://dx.doi.org/10.1145/2926718 The vector representation of the item i is computed on his neighborhood of length l. More two items share entities/property at a certain distance, more those items can be considered similar. State of the art Item neighborhood mapping
  • 44. 44 https://musiclynx.github.io/#/artist/ad79836d-9849-44df-8789-180bbc823f3c/Antonio%2520Vivaldi Alo Allik, Florian Thalmann, and Mark Sandler. 2018. MusicLynx: Exploring Music Through Artist Similarity Graphs. In The Web Conference 2018. Demo Track, pp 167-170. https://doi.org/10.1145/3184558.3186970 ● Access to different knowledge sources ● Maximum Degree Weighted (MDW): links to very large categories (i.e. Living People) are discouraged with respect to more significant ones. State of the art MusicLynx
  • 45. II.A Embeddings and Similarity pic https://pxhere.com/es/photo/1168914 Exploiting the Music knowledge
  • 46. 46 Word Embeddings (e.g. word2vec) corpus of document -> vectors that represent the semantic distribution of words in the text Graph Embeddings (e.g. node2vec) set of random walks -> vectors that represent the semantic distribution of entity in the graph Exploiting the Music knowledge Embeddings and Similarity II.A Main idea: nodes that occurs in similar contexts (neighborhood of nodes in a graph) are more similar, and will be closer in the vector space. Aditya Grover and Jure Leskovec. node2vec: Scalable Feature Learning for Networks. In 22nd ACM SIGKDD , 2016. 46
  • 47. Some problems: ● Our dataset was constantly growing ● The amount of nodes is huge ● Different purposes for recommendation: ○ radio broadcasting ○ concert programming ○ final users 47 computational-wise and time-wise expensive (multiple run of node2vec on huge amount of data) Exploiting the Music knowledge Embeddings and Similarity II.A
  • 48. 48 Compute embeddings at simple features level period of time musical key medium of performance genre ... Exploiting the Music knowledge Embeddings and Similarity II.A Solution
  • 49. 49 vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl Example: MoP The clarinet is more similar to the oboe or to the cello? vocabulary:iaml/mop/svc Exploiting the Music knowledge Embeddings and Similarity II.A
  • 50. 50 vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl vocabulary:iaml/mop/w vocabulary:iaml/mop/s vocabulary:iaml/mop/svc Example: MoP Exploiting the Music knowledge Embeddings and Similarity II.A Graph of vocabularies
  • 51. 51 vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl vocabulary:iaml/mop/svc Casting Detail Casting Detail Casting Detail Casting 862 times Casting 1213 times Example: MoP Exploiting the Music knowledge Embeddings and Similarity II.A Graph of usage
  • 52. 52 SPARQL endpoint subgraph (edgelist) selection of interesting properties (i.e. skos:broader) vectors embedding NODE2VEC s 1.34 0.98 0.20 w 1.44 1.21 0.31 svc 0.14 1.31 1.48 wcl -1.2 1.90 0.85 wob -0.83 2.32 1.03 Pasquale Lisena et al. Controlled Vocabularies for Music Metadata. 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, September 2018. Exploiting the Music knowledge Embeddings and Similarity II.A
  • 53. 53 vocabulary:iaml/mop/wob vocabulary:iaml/mop/wcl The clarinet is more similar to the oboe or to the cello? vocabulary:iaml/mop/svc 0.506 0.562 Exploiting the Music knowledge Embeddings and Similarity II.A Example: MoP
  • 54. 54 VECTOR SPACE OF MoPs ethnic chordophones ethnic flutes percussions brass orchestra woodwinds orchestra strings rare strings Exploiting the Music knowledge Embeddings and Similarity II.A
  • 55. 55 VECTOR SPACE OF GENRES Exploiting the Music knowledge Embeddings and Similarity II.A
  • 56. 56 Combine embeddings at complex features level artists works playlists Exploiting the Music knowledge Embeddings and Similarity II.A
  • 57. Example: Artists Exploiting the Music knowledge Embeddings and Similarity II.A
  • 58. 58 MOP embeddings: MOPGENRE KEY Artist’s features: BIRTH DATE DEATH DATE CASTING WORKS GENRE WORKS KEYS WORKS PLAYED MOP -0.02 0.01 0.01 0.00 -0.01 -0.02 0.01 0.00 -0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -0.00 0.00 0.00 0.07 -0.03 0.07 -0.02 -0.01 0.19 0.02 0.69 -0.19 -0.14 0.08 0.03 0.03 0.00 0.08 null null null null null -0.06 0.07 0.02 -0.03 0.00 Artist vector BIRTH PLACE DEATH PLACE FUNCTION FUNCTGeoNamesGeoNamesTime DIMENSIONALITY REDUCTION (PCA) Time AVG AVG AVG AVG AVG Exploiting the Music knowledge Embeddings and Similarity II.A Some data are unknown or not applicable null null null null null
  • 59. 59P. Lisena, R. Troncy (2017). Combining music specific embeddings for computing artist similarity. In 18th International Society for Music Information Retrieval Conference (ISMIR), Late-breaking & demo track. percentage of missing dimensions in artist 2 with respect to artist 1 Exploiting the Music knowledge Embeddings and Similarity II.A Example: Artists
  • 60. 60 Do all the properties have the same importance? Exploiting the Music knowledge Embeddings and Similarity II.A
  • 61. II.B Playlists and Weights pic:https://pixabay.com/photo-791076/ Exploiting the Music knowledge Which information is possible to extract from editorial playlists? RQ5 Idea: there are “unaware” rules that experts apply when realising a playlist.
  • 62. 62 ● Use playlists to give a weight to the influence of each dimension. ● No GOLD STANDARD available, creation of a “silver” one Radio France Playlists (50) Spotify Playlists (65) ITEMA3 Concerts (624) Philharmonie Concerts (186) ● Radio France Web Radio (7 channels) ● Realised by experts ● Classical section of Spotify app ● Realised by Spotify staff ● Real concerts that took place in Paris (studio + concert hall) Exploiting the Music knowledge Playlists and Weights II.B
  • 63. 63 variance within < variance between HOMOGENEOUS good for recommendation variance within > variance between INHOMOGENEOUS bad for recommendation Exploiting the Music knowledge Playlists and Weights II.B F test statistic = variance between / variance within Weights
  • 64. evaluation 64 Exploiting the Music knowledge Playlists and Weights II.B ● 7 music experts from partner institutions ● given the seed: ○ put bad items in the trash ○ sort according to preference
  • 65. evaluation 65 Exploiting the Music knowledge Playlists and Weights II.B The study of variance help us to identify which dimensions should be promoted for better recommendations.
  • 66. 66 Exploiting the Music knowledge Playlists and Weights II.B Under experimentation in live.philharmoniedeparis.fr
  • 67. The role of titles: Title2Rec Exploiting the Music knowledge “Relax Driving” Johannes Brahms Symphony n.3 “Beach Party” Luis Fonsi Despacito
  • 68. 68 Title2Rec: training Exploiting the Music knowledge Playlists and Weights II.B Content (id of tracks) Playlists SEQUENTIAL EMBEDDINGS (word2vec) CLUSTERS of playlists Titles DOCUMENTS fastText MODEL
  • 69. 69 yy :) christmas litmas guardians christmas christmas holiday christmas christmas the good stuff. xmas himym christmas pop xmas country happy holidays holidays christmas christmas hits 25 just cause stay christmas tis the season 🎄 christmas 🎄 christmas oldbutgold christmas christmas vibes christmas strong christmas winter wonderland christmas time december 15 xmas christmas christmas pop flight christmas deep christmas vibes christmas oldies work in progress christmas christmas playlist christmas music josh 🎄 christmas blah christmas & chill depression secret christmas christmas & chill christmas love :) christmas elite :) christmas special songs christmas christmas christmas jams jessica its lit classy pump up graduation at the moment .... christmas christmas christmas music good old days christmas mix christmas music 80s rock christmas 2015 xmas christmas christmas christmas christmas vibes 2017 songs christmas vibes!! christmas music holidays christmas 2016! christmas christmas club music summer 2015 christmasssss christmassss christmas christmas christmas christmas!! christmas christmas feels christmas christmas(:: christmas playlist great christmas playlist christmas & chill christmas christmas trap blast from the past christmas 2016 classics grad christmas christmas christmas christmas yessss christmas christmas rihanna christmas christmas songs christmas 2016!!!!! good vibes christmas christmas songs christmas christmas christmas favorites christmas christmas 2016 🎄 christmas last christmas christmas all my friends christmas christmas !! chirstmas the weeknd christmas 2015 christmas christmas lyrical party music wake up happy vibes 🎄 christmas calm country winter christmas christmas christmas pop christmas af ❄ christmas feel good :)) christmas christmas af christmas jams moana christmas merry christmas! christmas playlist christmas christmas silly love songs christmas </3 school 🎄 christmas christmas music christmas christmas music 🎄 christmas x-mas christmas bops christmas beachin' dance jamz christmas new wave its christmas christmas 🎄 christmas indie 2 christmas 1980 christmas jams christmas 2015 sunrise christmas christmas playlist christmas jams christmas white ella chirstmas sleep :))))) christmas random christmas dance christmas christmas december; christmas christmas favs christmas old christmas songs ~holidaze~ christmas christmas music xmas christmas holidays december christmas christmas christmas baby wedding music tis the season christmas relax holidays!! 🎅 🏼 christmas christmas christmas december '15 christmas!! christmas new songs christmas christmas Exploiting the Music knowledge Playlists and Weights II.B Title2Rec: training
  • 70. 70 Title2Rec: predicting Exploiting the Music knowledge Playlists and Weights II.B Given a new title: ● found the most similar titles among the known ones ● propose the most popular tracks among those titles Evaluated on Spotify’s Million Playlists Datasets in the context of the RecSys Challenge 2018 in the challenge: #37 over 112 #13 over 31
  • 71. II.C Learning MIDI Embeddings Exploiting the Music knowledge Is Graph representation also suitable for music content? RQ5
  • 72. 72 MIDI2vec Apply graph technologies to MIDI ● Transform MIDI flow in a graph ● Apply node2vec for learning graph embeddings Exploiting the Music knowledge Learning MIDI Embeddings II.C MIDI Group of Notes Pitch Duration Program Time Signature Tempo Velocity + + + + +
  • 73. 73 Experiment: genre and metadata prediction Dataset 1: SLAC 250 MIDI, balanced on 5/10 genres Accuracies on cross-fold validation: Dataset 2: MuseData 438 MIDI, unbalanced, linked to DOREMUS Accuracies on cross-fold validation: Exploiting the Music knowledge Learning MIDI Embeddings II.C Baseline: McKay et al (2010)
  • 75. 75 Which model best represents these rich data for final users and music scholars? DOREMUS model and Vocabularies What strategies to adopt for building a music Knowledge Graph? marc2rdf and other converters result: the DOREMUS Knowledge Graph How to make these data accessible to researchers and developers? SPARQL Transformer reshapes and merges the results for easy use RQ1 RQ2 RQ3 Main contributions
  • 76. 76 How can graph-based algorithms support music recommender systems? Embedding approach with generation and recombination of partial vectors Which information is possible to extract from editorial playlists? A study of editorial playlists, for weighting a recommender system Title2Rec: recommend music by the title of the playlist Graph representation is suitable also for music content? MIDI2vec: learning MIDI graph embeddings RQ4 RQ5 RQ6 Main contributions
  • 77. 77 Future Work (1/2) Short Term ● Studies on simplifications of the ontology (schema.org) ● Domain-based NLP for text-field information extraction Long Term ● Strategies for modeling librarian information representing meta-information on a 2nd level (RDF*) Modeling and accessing a KG
  • 78. 78 Future Work (2/2) Short term ● Split the dataset in historical period more precise training, faster performances ● Title2Rec + similarity-based recommender system application for editors ● Experiment MIDI embeddings on larger dataset Long term ● Gold standard dataset of classical music playlists ● Combining our strategy with more traditional ones (CF) ● MIDI ontology: extend and use in MIDI2vec Knowledge-aware Recommender system
  • 79. Publications Conference Poster&Demo Journal Tutorial Workshop EKAW'16 ISWC'16 EKAW'16 2016 ISWC'17 X2 ISMIR'17 K-CAP'17 DLfM'17 2017 ISWC'18 X2 ISMIR'18 ISMIR'18 BIBLIOTHEK - Forschung und Praxis ESWC'18 RecSys'18 TheWebConf'18 2018 ISWC'19 2019 PC Member ISWC’18 P&D, SAAM’18, DLfM’18, ISWC’19 P&D, K-CAP’19 as sub-reviewer: KAARS’18, TheWebConf’19 Student Supervision 2 Master Thesis supervisions 10 Semester Projects supervisions Lecturer for WebInt and Aalto BootCamp Talks Des Catalogues au Web des Données - BnF, Paris Classical Music and Knowledge Graphs - Semantic Web course, PoliTo - WAI meeting, VU Amsterdam - Research seminar, Deezer, Paris
  • 80. 80 References (1/2) ● M. Schedl (2015) Towards Personalizing Classical Music Recommendations. 2015 IEEE International Conference on Data Mining Workshop (ICDMW), Atlantic City, NJ, USA, pp. 1366-1367. ● Y. Raimond, S. Abdallah, M. Sandler, and F. Giasson (2007). The Music Ontology. In 15th International Conference on Music Information Retrieval (ISMIR). 417–422 ● P. Choffé and F. Leresche (2016). DOREMUS: connecting sources, enriching catalogues and user experience. In 24th IFLA World Library and Information Congress. ● P. Lisena et al. (2018). Controlled Vocabularies for Music Metadata. In 19th International Conference on Music Information Retrieval (ISMIR). Paris, France. ● P. Lisena et al. (2017) Modeling the Complexity of Music Metadata in Semantic Graphs for Exploration and Discovery. In 4th International Workshop on Digital Libraries for Musicology (DLfM’17), Shanghai, China. ● Booth et al. (2019) Toward Easier RDF. In W3C Workshop on Web Standardization for Graph Data. ● Lisena P. et al. (2019). Easy Web API Development with SPARQL Transformer. In ISWC’19. ● S. Oramas, V. C. Ostuni, T. Di Noia, X. Serra, and E. Di Sciascio. Sound and music recommendation with knowledge graphs. ACM Trans. Intell. Syst. Technol. 8, 2, Article 21 (October 2016), 21 pages.
  • 81. 81 References (2/2) ● Alo Allik, Florian Thalmann, and Mark Sandler (2018). MusicLynx: Exploring Music Through Artist Similarity Graphs. In The Web Conference 2018. Demo Track, pp 167-170. ● Aditya Grover and Jure Leskovec. (2016) node2vec: Scalable Feature Learning for Networks. In 22nd ACM SIGKDD. ● McKay, C., Burgoyne, J., Hockman, J., B. L. Smith, J.,Vigliensoni, G., and Fujinaga, I. (2010). Evaluating the Genre Classification Performance of Lyrical Features Relative to Audio, Symbolic and Cultural Features. In ISMIR 2011, Utrecht, The Netherlands ● Meroño-Peñuela, A., Hoekstra, R., Gangemi, A., Bloem, P., de Valk, R., Stringer, B., Janssen, B., de Boer, V.,Allik, A., Schlobach, S., et al. (2017). The MIDI Linked Data Cloud. In ISWC 2017, Vienna, Austria. ● Huang, A. and Wu, R. (2016). Deep Learning for Music. Computing Research Repository (CoRR), https://arxiv.org/abs/1606.04930 . ● Peter Knees and Markus Schedl (2013). A survey of music similarity and recommendation from music context data. ACM Trans. Multimedia Comput. Commun. Appl. 10, 1, Article 2 (December 2013), 21 pages. ● Palumbo, Rizzo, Troncy. (2017) entity2rec: Learning user-item relatedness from knowledge graphs for top-N item recommendation. In RECSYS 2017, Como, Italy.