Web-Instrumented Man-Machine Interactions Research

WIMMICS
Fabien GANDON, @fabien_gandon http://fabien.info
   
Web-Instrumented Man-
Machine Interactions,
Communities and Semantics

WIMMICS TEAM
▪ Inria
▪ CNRS
▪ University Côte D’Azur (UCA)
I3S
Web-Instrumented Man-Machine Interactions,
Communities and Semantics

MULTI-DISCIPLINARY TEAM
▪ 35~55 members
▪ ~15 nationalities
▪ 1 DR, 4 Professors
▪ 3CR, 3 Assistant professors
DR/Professors:
▪ Fabien GANDON, Inria, AI, KRR, Semantic Web, Social Web, K. Graphs
▪ Nhan LE THANH, UCA, Logics, KR, Emotions, Workflows, K. Graphs
▪ Peter SANDER, UCA, Web, Emotions
▪ Andrea TETTAMANZI, UCA, AI, Logics, Evo, Learning, Agents, K. Graphs
▪ Marco WINCKLER, UCA, Human-Computer Interaction, Web, K. Graphs
CR/Assistant Professors:
▪ Michel BUFFA, UCA, Web, Social Media, K. Graphs
▪ Elena CABRIO, UCA, NLP, KR, Linguistics, Q&A, Text Mining, K. Graphs
▪ Olivier CORBY, Inria, KR, AI, Sem. Web, Programming, K. Graphs
▪ Catherine FARON-ZUCKER, UCA, KR, AI, Semantic Web, K. Graphs
▪ Damien GRAUX, Inria, Linked Data, Sem. Web, Querying, K. Graphs
▪ Serena VILLATA, CNRS, AI, Argumentation, Licenses, Rights, K. Graphs
Research engineer: Franck MICHEL, CNRS, Linked Data, Integration, DB, K. Graphs
External:
▪ Andrei Ciortea (University of St. Gallen) Agents, WoT, Sem. Web, K. Graphs
▪ Nicolas DELAFORGE (Mnemotix) Sem. Web, KM, Integration, K. Graphs
▪ Alain GIBOIN, (Retired CR Inria), Interaction Design, KE, User & Task, K. Graphs
▪ Freddy LECUE (Thales, Montreal) AI, Logics, Mining, Big Data, S. Web , K. Graphs

CHALLENGE
to bridge social semantics and
formal semantics on the Web

CHALLENGES
typed graphs to analyze,
model, formalize and
implement social semantic
web applications for
epistemic communities
 multidisciplinary approach for analyzing and modeling
▪the many aspects of intertwined information systems
▪communities of users and their interactions
 formalizing and reasoning on these models using typed graphs
▪new analysis tools and indicators
▪new functionalities and better management

WEB GRAPHS
(meta)data of
the relations
and the
resources of the
web
…sites …social …of data …of services
+ + + +…
web…
= +
…semantics
+ + + +…
= +
typed
graphs
web
(graphs)
networks
(graphs)
linked data
(graphs)
workflows
(graphs)
schemas
(graphs)

URI, IRI, URL, HTTP URI
CONTRIBUTE TO DATA AND SCHEMATA STANDARDS ON THE WEB
JSON
RDF
JSON LD
N-Triple
N-Quad
Turtle/N3
TriG
RDFS
OWL
SPARQL
XML
HTML
RDF XML
HTTP
Linked Data
CSV-LD R2RML
GRDDL
RDFa
SHACL
LDP

The four research axes of Wimmics
-
contributing to research in AI and
Semantic Web along 4 axes:
1. Web-based user modeling and interaction
design
2. Social interactions and content analysis on
the Web
3. Knowledge extraction and representation
for and by linked data on the Web
4. Web-oriented and Web-dedicated artificial
intelligence algorithms


G2 H2

G1 H1
<
Gn Hn




e.g. cultural data is a weapon of mass construction

PUBLISHING
▪ extract data (content, activity…)
▪ provide them as linked data
DBPEDIA.FR (extraction, end-point)
180 000 000 triples
models
Web architecture
[Cojan, Boyer et al.]

PUBLISHING
DBpedia.fr usage
number of queries per day
70 000 on average
2.5 millions max
185 377 686 RDF triples extracted and mapped
public dumps, endpoints, interfaces, APIs…

PUBLISHING
DBpedia.fr active since
2012

REUSE
▪ build and help build applications DBPEDIA.FR (extraction, end-point)
180 000 000 triples
Zone 47
BBC
HdA Lab
RAWebSem
MCC Contest
MonaLIA

James Bridle’s twelve-volume encyclopedia of all
changes to the Wikipedia article on the Iraq War
booktwo.org

EXTRACTED
entire edition history as
linked open data
1.9 billion triples describing the 107 million revisions since the first page was created
<http://fr.wikipedia.org/wiki/Victor_Hugo> a prov:Revision ;
dc:subject <http://fr.dbpedia.org/resource/Victor_Hugo> ;
swp:isVersion "3496"^^xsd:integer ;
dc:created "2002-06-06T08:48:32"^^xsd:dateTime ;
dc:modified "2015-10-15T14:17:02"^^xsd:dateTime ;
dbfr:uniqueContributorNb 1295 ;
(...)
dbfr:revPerYear [ dc:date "2015"^^xsd:gYear ; rdf:value
"79"^^xsd:integer ] ;
dbfr:revPerMonth [ dc:date "06/2002"^^xsd:gYearMonth ;
rdf:value "3"^^xsd:integer ] ;
(...)
dbfr:averageSizePerYear [ dc:date "2015"^^xsd:gYear ;
rdf:value "154110.18"^^xsd:float
] ;
dbfr:averageSizePerMonth [ dc:date
"06/2002"^^xsd:gYearMonth ;
rdf:value "2610.66"^^xsd:float ]
;
(...)
dbfr:size "159049"^^xsd:integer ;
dc:creator [ foaf:nick "Rinaldum" ] ;
sioc:note "wikification"^^xsd:string ;
prov:wasRevisionOf <http:// … 119074391> ;
prov:wasAttributedTo [ foaf:name "Rémih" ; a prov:Person,
foaf:Person ] .
<http:// … 119074391> a prov:Revision ;
dc:created "2015-09-29T19:35:34"^^xsd:dateTime ;
dbfr:size "159034"^^xsd:integer ;
dbfr:sizeNewDifference "-5"^^xsd:integer ;
sioc:note "/*Années théâtre*/ neutralisation"^^xsd:string ;
prov:wasAttributedTo [ foaf:name "Thouny" ; a prov:Person,
foaf:Person ] ;
prov:wasRevisionOf <http://... 118903583> .
(...)
<http:// … oldid=118201419> a prov:Revision ;
prov:wasAttributedTo [ foaf:name "OrlodrimBot" ; a
prov:SoftwareAgent ] ;
(...)
[Gandon, Boyer, Corby, Monnin 2016]

DEMO
Facetted history portals
Elections
in France

DEMO
Elections
in France
Death of
C. Lee

DEMO
Elections
in France
Death of
C. Lee
Events in
Ukraine

DBPEDIA & STTL
declarative transformation
language from RDF to text
formats (XML, JSON, HTML,
Latex, natural language, GML,
…) [Cojan, Corby, Faron-Zucker et al.]

COVID ON THE WEB
[Corby, Michel, Gazzotti, Winckler, et al. 2019]
▪ integrate multiple datasets in heterogeneous formats
▪ perform information extraction to enrich
▪ perform inferences and validation to improve
▪ provide a public end-point for reuse
▪ provide querying and visualization services

vs. use cases…
• Scenario 1: Help clinicians analyze clinical trials and take evidence-based decisions
• Scenario 3: Help missions heads from Cancer Institute elaborate research programs
to study the links between cancer and coronavirus
121
[Giboin, et al.]

COVID ON THE WEB
RDF
translator
Jupyter Notebook
Python, R & analytics
Corese
engine
query
&
infer
ACTA Web application
visualization of argument graphs
Corese portal
Data browsing
MGExplorer
Data visualization
Open Data publication
Zenodo, Github, Virtuoso
Named
Entities
extractor
s
Covid-on-the-Web
dataset
LOD
ACTA pipeline
extraction of argument graphs
1
1
2
2
1
3
3
3
vocabularies
& datasets
3
3
3
2
Covid-19
Open
Research
Dataset
Process data &
derive “smarter” data
Means to exploit data
[Michel, Gazzotti, Gandon et al.]

COVID ON THE WEB
Biomedical
researchers
& managers
Data analysts
…
RDF
translator
Jupyter Notebook
Python, R & analytics
Corese
engine
query
&
infer
ACTA Web application
visualization of argument graphs
Corese portal
Data browsing
MGExplorer
Data visualization
Open Data publication
Zenodo, Github, Virtuoso Applications
Named
Entities
extractor
s
dereference, query, download
query, browse, analyze, make sense
Covid-on-the-Web
dataset
LOD
ACTA pipeline
extraction of argument graphs
1
1
2
2
1
3
3
3
vocabularies
& datasets
3
3
3
2
Covid-19
Open
Research
Dataset
Process data &
derive “smarter” data
Means to exploit data Biomedical research
[Michel, Gazzotti, Gandon et al.]

Dataset description No. RDF triples
dataset description + definition of a few properties 170
articles metadata (title, authors, DOIs, journal etc.) 3 722 381
named entities identified by Entity-fishing in articles titles/abstracts 35 049 832
named entities identified by Entity-fishing in articles bodies 1 156 611 321
named entities identified by Bioportal Annotator in articles titles/abstracts 104 430 547
named entities identified by DBpedia Spotlight in articles titles/abstracts 65 359 664
argumentative components and PICO elements by ACTA from articles titles/abstracts 7 469 234
Total 1 361 451 364
125

more data, more usages, more users

NEW USAGES
e.g. e-learning & serious games
[Rodriguez-Rocha, Faron-Zucker et al.]

QUIZZES
Automated generation of
quizzes
[Rodriguez-Rocha, Faron-Zucker et al.]
rdf:type
p
rdfs:subClassOf
?
?
?
?
RDF Q&A
Generation
NL Questions
Generation
Ranking
Question
SELECTION
Knowledge
Graph
Quiz
Q&A

PREDICT STUDENTS
▪ a model of the students' learning
▪ predict success or failure to questions
▪ features from KG representations
▪ Logistic Regression (LR) / Factorization Machines
(FM) / Deep Factorization Machines (DeepFM)
[Rodriguez-Rocha, Faron, Ettorre, Michel et al. 2020]
Answers
Questions
s: students identifiers
q: questions identifiers
r: responses identifiers
a: number of attempts
w: number of wins
T: questions text embeddings
Q: graph embeddings of the questions
R: graph embeddings of the answers
e: extra group of calculated features:
question_difficulty,student_ability,
student_ability_progressive,
student_ability_progressive_question_difficulty
Features

EDUMICS
▪ Ontology EduProgression: OWL modeling of scholar program
▪ Ontology RefEduclever: new education referential for Educlever
▪ Migration and persistence in graph databases
▪ Reasoning, query, interactions, recommendation
[Fokou, Faron et al. 2017]

“
Smarter Cities – IBM Dublin
[Lécué, 2015]

QUESTION ROUTING
▪ emails to the customer service (eg 350000/day “Crédit Mutuel”)
▪ detect topics in order to “understand” a question
▪ 3 humans annotate 142 questions (Krippendorff’s Alpha 0,70)
▪ NLP and semantic processing for features extraction
▪ ML performance comparison for question classification
Naive Bayes, Sequential Minimal Optimisation (SMO),
Random Forest, RAndom k-labELsets (RAkEL)
[Gazzotti, et al. 2017]
NE
recognition
(L,T)
Removing
special
characters
Tokenization
(L,T)
Spell
Checking
(L,T)
Lemmatization
(L)
Vector
generation
BOW/N-gram
Replacement in documents
Consider as feature
Input
Document
ML
workflow
L: Language dependent - T: Text dependent
Unbalanced Topics
Metrics uni uni⨁bi uni+bi+tri uni⨁NE syn syn⨁hyper syn⨁NE
Hamming Loss 0,0381 0,0370 0,0374 0,0373 0,0399 0,0412 0,0405

integrating heterogenous sources

SCIENTIFIC HERITAGE
▪ TAXREF Vocabulary
▪ Data extraction and
publication
[Tounsi, Callou, Michel, Pajo, Faron Zucker et al.]

rr:objectMap
1
1
0-1
0-1
1
0-1
0-1
0-1
0-1
1
1
rr:GraphMap
rr:graphMap
0-1
xrr:logicalSource
xrr:LogicalSource
xrr:query
Query String
rml:iterator Iteration pattern
rr:IRI, rr:BlankNode,rr:Literal,
xrr:RdfList, xrr:RdfBag,
xrr:RdfSeq, xrr:RdfAlt
reference expr.
xrr:nestedTermMap
xrr:NestedTermMap
rr:inverseExrpression
xrr:reference
reference expr.
reference expr.
rr:ObjectMap
HETEROGENEITY
xR2RML mapping language
and SPARQL query rewriting
[Michel et al.]
<AbstractQuery> ::= <AtomicQuery> | <Query> |
<Query> FILTER <SPARQL filter> | <Query> LIMIT <integer>
<Query> ::= <AbstractQuery> INNER JOIN <AbstractQuery> ON {v1, … vn} |
<AtomicQuery> AS child INNER JOIN <AtomicQuery> AS parent
ON child/<Ref> = parent/<Ref> |
<AbstractQuery> LEFT OUTER JOIN <AbstractQuery> ON {v1, … vn} |
<AbstractQuery> UNION <AbstractQuery>
<AtomicQuery> ::= {From, Project, Where, Limit}
<Ref> ::= a valid xR2RML data element reference

µSERVICES
Linked Data access to Web APIs.
[Michel et al.]
SPARQL Client
Service Logics
Web API
JSON-LD
Profile
SPARQL
INSERT/CONSTR
HTTP
query
JSON
response
Triple
store
SPARQL Micro-Service
(1)
(4) (2)
(3)
LD Client Web Server
(1’)
(4’)
http://example.org/photo/472495

LD µSERVICES
APIs as linked data
[Michel , et al.]
SPARQL micro-
service
SPARQL SD
graph
Shapes
graph
SPARQL engine
Web
API
(5)

LD µSERVICES
APIs as linked data
[Michel , et al.]
HTML
JDON-LD
</>
SPARQL micro-
service
(1)
(4) SPARQL query
LD-based
application
SPARQL SD
graph
Shapes
graph
SPARQL engine
Web
API
(5)

LD µSERVICES
APIs as linked data
[Michel , et al.]
HTML
JDON-LD
</>
SPARQL micro-
service
(1)
(4) SPARQL query
LD-based
application
SPARQL SD
graph
Shapes
graph
SPARQL engine
Web
API
(5)
&

UNCERTAINTY
▪ Representing uncertainty theories
▪ Publishing it with linked data
▪ Negotiating the theory over HTTP
▪ Combining uncertainty statements
[Djebri, Tettamanzi, Gandon, 2019]

UNCERTAINTY
publishing theories and calculi as linked data
prob:Probability a munc:UncertaintyApproach;
munc:hasUncertaintyFeature prob:probabilityValue;
munc:hasUncertaintyOperator prob:and.
prob:probabilityValue prob:and prob:multiplyProbability.
prob:Probability prob:probabilityValue
prob:and
ex:multiplyProbability
munc:hasUncertainty
Feature
munc:hasUncertainty
Operator

UNCERTAINTY
publishing theories and calculi as linked data
function prob:multiplyProbability(?s1, ?s2, ?c) {
let(?v1 = munc:getMeta(?s1, prob:Probability)){
if(prob:verifyIndependent(?s1, ?s2) == true)
?v2 = munc:getMeta(?s2, prob:Probability, xt:list(?s1, ?c))
return (?v1 * ?v2)
} else {
?v2 = munc:getMeta(?s2, prob:Probability)
return (?v1 * ?v2)
}
}
}
prob:Probability a munc:UncertaintyApproach;
munc:hasUncertaintyFeature prob:probabilityValue;
munc:hasUncertaintyOperator prob:and.
prob:probabilityValue prob:and prob:multiplyProbability.
prob:Probability prob:probabilityValue
prob:and
ex:multiplyProbability
munc:hasUncertainty
Feature
munc:hasUncertainty
Operator

UNCERTAINTY
translate and negotiate theories
[Djebri et al 2019]
• Specify uncertainty in parameter linked to the format
• GET /some/resource HTTP/1.1
Accept:
text/turtle;uncertainty="http://example.com/Probability";q=0.8,
text/turtle;uncertainty="http://example.com/Possibility";q=0.2;
• Use uncertainty as a profile : prof-Conneg
Accept: text/turtle;q=0.8;profile="prob:Probability",
text/turtle;q=0.2;profile="poss:Possibility"
• HEAD /some/resource HTTP/1.1
Accept: text/turtle;q=0.9,application/rdf+xml;q=0.5
Link: <http://example.com/Probability>; rel="profile" (RFC 6906)

MoReWAIS Mobile Read Write Access and Intermittent to Semantic Web
France (Wimmics, Inria) – Senegal (LANI, UGB Saint-Louis) Project
explore the specificities (advantages and constraints) of mobile P2P knowledge
sharing and addressing its limitations (e.g. intermittent access, limited resources)
[Toure et al.]
&

MoRAI: Geographic and Semantic Overlay Network
• Three-level P2P architecture : mobile peers, super-peers and remote sources
• Random Peer Sampling (RPS) overlay +
Semantic Overlay Network (SON) +
Geographic Overlay Network (GON)
• Experimental validation/simulation
[Toure et al.]

CRAWLING
▪ Predict data availability
▪ Select features of URIs
▪ Learn crawling selection
(KNN/NaiveBayes/SVM)
▪ Online learning w. crawling
(FTRL-proximal algorithm)
[Huang, Gandon 2019]

QUERY
• automatically suggest relevant data sources to solve a query
• sets of path features: star, sink, chain
• approximate containment search: locality sensitive hashing
[Huang, Gandon 2020]

“searching” comes in many flavors

SEARCHING
▪ exploratory search
▪ question-answering
DBPEDIA.FR (SPARQL end-point)
180 000 000 triples

SEARCHING
180 000 000 triples
DISCOVERYHUB.CO
semantic spreading
activation
new evaluation protocol
[Marie, Giboin, Palagi et al.]

SEARCHING
180 000 000 triples
DISCOVERYHUB.CO
semantic spreading
activation
new evaluation protocol
[D:Work], played by [R:Person]
[D:Work] stars [R:Person]
[D:Work] film stars [R:Person]
starring(Work, Person)
linguistic relational
pattern extraction
named entity recognition
similarity based SPARQL
generation
select * where {
dbpr:Batman_Begins dbp:starring ?v .
OPTIONAL {?v rdfs:label ?l
filter(lang(?l)="en")} }
[Cabrio et al.]
[Marie, Giboin, Palagi et al.]
QAKiS.ORG

SEARCHING
e.g. DiscoveryHub
exploratory search
[Marie, Gandon, Ribière.]

semantic spreading activation
SIMILARITY
FILTERING
discoveryhub.co
[Marie, Gandon, Ribière.]

SEARCHING
e.g. QAKIS
question-answering
[Cabrio et al.]

learning linguistic patterns of queries
[Cabrio et al.]

MULTIMEDIA
answer visualization
through linked data
[Cabrio et al.]

BROWSING
e.g. SMILK plugin
[Lopez, Cabrio, et al.]

BROWSING
e.g. SMILK plugin
[Nooralahzadeh, Cabrio, et al.]

SEARCHING
e.g. DiscoveryHub
exploratory search

relevant
not known
known
not relevant
EVALUATING
user-centric studies

INTERACTION
design and evaluation
Favoris
Nouvelle recherche TEMPS
Debut test Free Jazz 24s
Free improvisation 33s
(fiche) Avant-garde 47s
John Coltrane (vidéo) 1min 28
Marc Ribot 2min11
(fiche) experimental music 2min18 2min23
Krautrock 2min31
(fiche) Progressive rock 2min37 2min39
Red (King Crimson album) 2m52 2min59
King
Crimson 3min05
(fiche) Jazz fusion 3min18
(fiche) Free Jazz 3min32 3min54
Sun Ra 4min18
(fiche) Hard bop 4min41 4min47
Charles
Mingus (vidéo) 5min29
(fiche) Third Stream (vidéo) 6min20
Bebop 7min19
Modal jazz 7min26
(fiche) Saxophone 7min51 7min55
Mel Collins
21st CenturySchizoid Band
Crimson Jazz Trio
(fiche)
King
Crimson
(fiche)
Robert
Fripp
Miles Davis
Thelonious Monk
(fiche) Blue Note Record
McCoy Tyner
(fiche) Modal Jazz
(fiche) Jazz
Chick Corea
(fiche) Jazz Fusion
Return to Forever
MahavishnuOrchestra
Shakti (band)
U.Srinivas
Bela Fleck
Flecktones
John McLaughlin (musician)
Dixie Dregs
FICHE Dixie Degs
T Lavitz
Jordan Rudess
Behold… The Arctopus
(fiche) Avant-garde metal
Unexpected
FICHE unexpected
Dream Theater
King
Crimson
(fiche) Jazz fusion
King
Crimson
TonyLevin
(fiche) Anderson Bruford Wakeman Howe
(fiche) Rike Wakeman (vidéo)
Fin test
[Palagi, Marie, Giboin et al.]

(RE)DESIGN
interface evolutions
[Palagi, Marie, Giboin et al.]

METHODS & CRITERIA
▪ interaction design and evaluation
▪ exploratory search process model
[Palagi, Giboin et al. 2018]
A. Define the search space
B. Query (re)formulation
C. Information gathering
D. Put some information aside
E. Pinpoint search
F. Change of goal(s)
G. Backward/forward steps
H. Browsing results
I. Results analysis
J. Stop the search session
Previous features Feature Next features
NA A B ; J
A ; F B G ; H ; I ; J
D ; E ; I C D ; E ; F ; G ; H ; J
E ; I D C ; F ; G ; J
G ; H ; I E C ; D ; F ; G ; J
C ; D ; E ; G ; H ; I F B ; H ; I ; J
B ; D ; E ; H ; I G E ; F ; H ; I ; J
B ; F ; G ; I H E ; F ; G ; ; I ; J
B ; F ; G ; H I C ; D ; E ; F ; G ; H ; J
all J NA

CHEXPLORE
Plugin to evaluate the
interface of an exploratory
search engine
[Palagi, Giboin et al. 2018]

“
« a Web-Augmented Interaction (WAI) is a
user’s interaction with a system that is
improved by allowing the system to
access Web resources »
[Gandon, Giboin, WebSci17]

AZKAR
remotely visit and interact
with a museum through a
robot and via the Web
[Buffa et al.]

ALOOF: Web and Perception
[Cabrio, Basile et al.]
Semantic Web-Mining and Deep Vision for Lifelong Object Discovery (ICRA 2017)
Making Sense of Indoor Spaces using Semantic Web Mining and Situated Robot Perception (AnSWeR 2017)

ALOOF: robots learning by reading on the Web
Annie cuts the bread in the kitchen with her knife dbp:Knife aloof:Location dbp:Kitchen

First Object Relation Knowledge Base:
46.212 co-mentions gave 49 tools, 14
rooms, 101 “possible location” relations,

▪ First Object Relation Knowledge Base: 46212 co-mentions, 49 tools, 14 rooms, 101
“possible location” relations, 696 tuples <entity, relation, frame>
▪ Evaluation: 100 domestic instruments, 20 rooms, 2000 crowdsourcing judgements
▪ Shared between robots through a shared Web knowledge base
[Cabrio, Basile
et al. 2017]

ALOOF: RDF dataset about objects
▪ common sense knowledge about objects: classification, prototypical locations
and actions
▪ knowledge extracted from natural language parsing, crowdsourcing,
distributional semantics, keyword linking, ...

WASABI
Web-augmented music
interactions
[Buffa et al.]

WASABI
Web-augmented music
interactions
[Buffa et al.]
[Fell et al.]

DEBATES & EMOTIONS
#IRC
[Villata, Cabrio et al.]

DEBATES & EMOTIONS
#IRC argument rejection
attacks-disgust
[Villata, Cabrio et al.]

SENTIMENTS
▪ sentiment recognition in search results
▪ Automatic detection of affect
▪ Automatically interplay emotions to sentiment polarity
▪ Broader sense of sentiment ( Valence, Arousal, Dominance)
▪ Use case: Brexit scenario
▪ Propaganda detection based on argumentative techniques
[Vorakitphan, Cabrio, Villata 2020]

OPINIONS
NLP, ML and arguments
to monitor online image
[Villata, Cabrio, et al.]

ARGUMENT MINING ON
CLINICAL TRIALS
▪ NLP, ML and arguments
▪ assist evidence-based medicine
▪ support doctors and clinicians
▪ identify doc. for certain disease
▪ analyze argumentative content
and PICO elements
[Mayer, Cabrio, Villata]

ARGUMENT MINING ON
POLITICAL SPEECHES
▪ NLP and Machine Learning.
▪ Support historians/social science scholars
▪ Analyze arguments in political speeches
▪ DISPUTool : 39 political debates,
last 50 years of US presidential
campaigns (1960-2016)
[Mayer, Cabrio, Villata]

CYBERBULLYING
CREEP EU project: detect and prevent
[Corazza, Arslan, Cabrio, Villata]

QUERY & INFER
▪ graph rules and queries
▪ deontic reasoning
▪ induction
CORESE
 &
G2 H2
 &
G1 H1
<
Gn Hn
abstract graph machine
STTL
[Corby, Faron-Zucker et al.]

QUERY & INFER
▪ induction
CORESE
 &
G2 H2
 &
G1 H1
<
Gn Hn
RATIO4TA
predict &
explain
STTL
[Hasan et al.]

QUERY & INFER
▪ induction
CORESE
INDUCTION
 &
G2 H2
 &
G1 H1
<
Gn Hn
RATIO4TA
predict &
explain
find missing
knowledge
STTL
[Hasan et al.]
[Tettamanzi
et
al.]

QUERY & INFER
▪ induction
CORESE
LICENTIA
INDUCTION
 &
G2 H2
 &
G1 H1
<
Gn Hn
RATIO4TA
predict &
explain
find missing
knowledge
deontic reasoning, license
compatibility and composition
STTL
[Hasan et al.]
[Tettamanzi
et
al.]
[Villata et al.]

QUERY & INFER
e.g. CORESE/KGRAM
[Corby et al. since 1999]

FO → R  GF  GR
mapping modulo an ontology
car
vehicle
car(x)vehicle(x)
GF
GR
vehicle
car
O
RIF-BLD SPARQL RIFSPARQL
?x ?x
C C
List(T1. . . Tn) (T1’. . . Tn’)
OpenList(T1. . . Tn T)
External(op((T1. . . Tn))) Filter(op’ (T1’. . . Tn’))
T1 = T2 Filter(T1’ =T2’)
X # C X’ rdf:type C’
T1 ## T2 T1’ rdfs:subClassOf T2’
C(A1 ->V1 . . .An ->Vn)
C(T1 . . . Tn)
AND(A1. . . An) A1’. . . An’
Or(A1. . . An) {A1’} …UNION {An’}
OPTIONAL{B}
Exists ?x1 . . . ?xn (A) A’
Forall ?x1 . . . ?xn (H)
Forall ?x1 . . . ?xn (H:- B) CONSTRUCT { H’}
WHERE{ B’}
restrictions
equivalence no equivalence
extensions

FO → R  GF  GR
car
vehicle
car(x)vehicle(x)
GF
GR
vehicle
car
O
truck
car
 
 
 





=



1
2
1 ,
, )
(
2
1
2
1
2
2
1
2
1
)
,
(
let
;
)
,
( t
t
t
t
t t
depth
H
c t
t
l
t
t
H
t
t c
 ( )
)
,
(
)
,
(
min
)
,
(
let
)
,
( 2
1
,
2
1
2
2
1 2
1
t
t
l
t
t
l
t
t
dist
H
t
t c
c H
H
t
t
t
t
c +
=

 

vehicle
car
O
truck
t1(x)t2(x) → d(t1,t2)< threshold

Lymphoma
Cancer
Lymphoma(x) Cancer(x)
GD
GQ
Cancer
Lymphoma
O
217

FO → R  GD  GQ
Lymphoma
Cancer
Lymphoma(x) Cancer(x)
GD
GQ
Cancer
Lymphoma
O
[Corby et al.]
AI methods: knowledge graphs, ontology-based formalisms, querying, validating and reasoning
218

Query for co-occurrence of cancers & coronaviruses with R Jupyter Notebooks…
219

Visualisation pipeline
[Menin, Winckler, Corby, Giboin, Faron et al. 2020]
220
?

LDSCRIPT
a Linked Data Script Language
FUNCTION us:status(?x) {
IF (EXISTS { ?x ex:hasSpouse ?y }||EXISTS { ?y ex:hasSpouse ?x },
ex:Married, ex:Single) }
[Corby, Faron Zucker, Gandon, ISWC 2017]

SPARQL ENDPOINT ACCESS CONTROL
Protect SPARQL endpoint from hostile actions
Set of protected Features:
SPARQL Update, Load RDF data, Service clause
Set of Access Rights:
PUBLIC, PROTECTED, PRIVATE
Assign Access Rights to Features:
SPARQL Update -> PRIVATE
Service <http://fr.dbpedia.org> -> PROTECTED
Assign Access Right to User Action:
User query -> PUBLIC
PUBLIC action cannot access PRIVATE Feature
[Corby, 2021]

RDF & SPARQL ACCESS CONTROL
Assign Access Right to RDF triples and SPARQL
Queries
• SPARQL Query has access to subset of RDF triples
• RDF Graph extended with Access Rights
• SPARQL Interpreter extended with Access Rights
e.g.
Assign Access Rights to RDF triples according to URIs or namespaces
URI foaf:address -> PRIVATE
Namespace foaf: -> PUBLIC
select * where { ?x ?p ?y } -> PUBLIC
Query can access PUBLIC foaf:name
Query cannot access PRIVATE foaf:address
[Corby, 2021]

DISTRIBUTED
inductive index creation for a
triple store [Basse, Gandon, Mirbel]

DISTRIBUTED
Querying heterogeneous and
distributed data [Gaignard,Corby et al.]

This project has received funding from the European Union's Horizon 2020
research and innovation programme under grant agreement 825619.
ONTOLOGY FOR AI ITSELF
▪ ontology and metadata of AI resources
▪ SHACL to validate AI4EU these RDF graphs
▪ online endpoint http://corese.inria.fr
▪ predefined SPARQL queries, SHACL shapes, display
[Corby et al., 2019]

mining interesting association rules
AI methods: clustering + community detection + dimensionality
reduction (auto-encoder) + Frequent Pattern Growth
[Cadorel, Tettamanzi]
241
[WI-IAT 2020]

• hidden patterns to enrich the dataset
• novel hypotheses for biomedical research
242
[WI-IAT 2020]

• hidden patterns to enrich the dataset
• novel hypotheses for biomedical research
• error detection in the dataset
• relevant clusters & communities for navigation
243
[WI-IAT 2020]

Visualization of Association found in Covid dataset
[Menin, Cadorel, Tettamanzi, Winckler]
244

DEONTICS
e.g. Licencia
[Villata et al.]

DEONTICS
Legal Rules on the Semantic Web
OWL + Named Graphs + SPARQL Rules
Named Graph (state of affair) Subject Predicate Object
http://ns.inria.fr/nrv-inst#StateOfAffairs1 Tom http://ns.inria.fr/nrv-inst#activity driving at 100km/h
http://ns.inria.fr/nrv-inst#StateOfAffairs1 Tom http://www.w3.org/2000/01/rdf-schema#label Tom
http://ns.inria.fr/nrv-inst#StateOfAffairs1 can't drive over 90km http://www.w3.org/1999/02/22-rdf-syntax-ns#type violated requirement
http://ns.inria.fr/nrv-inst#StateOfAffairs1 can't drive over 90km has for violation http://ns.inria.fr/nrv-inst#StateOfAffairs1
http://ns.inria.fr/nrv-inst#StateOfAffairs1 driving at 100km/h http://ns.inria.fr/nrv-inst#speed 100
http://ns.inria.fr/nrv-inst#StateOfAffairs1 driving at 100km/h http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://ns.inria.fr/nrv-inst#Driving
http://ns.inria.fr/nrv-inst#StateOfAffairs1 driving at 100km/h http://www.w3.org/2000/01/rdf-schema#label "driving at 100km/h"@en
Named Graph (state of affair) Subject Predicate Object
http://ns.inria.fr/nrv-inst#StateOfAffairs2 Jim http://ns.inria.fr/nrv-inst#activity driving at 90km/h
http://ns.inria.fr/nrv-inst#StateOfAffairs2 Jim http://www.w3.org/2000/01/rdf-schema#label Jim
http://ns.inria.fr/nrv-inst#StateOfAffairs2 can't drive over 90km http://www.w3.org/1999/02/22-rdf-syntax-ns#type compliant requirement
http://ns.inria.fr/nrv-inst#StateOfAffairs2 can't drive over 90km has for compliance http://ns.inria.fr/nrv-inst#StateOfAffairs2
http://ns.inria.fr/nrv-inst#StateOfAffairs2 driving at 90km/h http://ns.inria.fr/nrv-inst#speed 90
http://ns.inria.fr/nrv-inst#StateOfAffairs2 driving at 90km/h http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://ns.inria.fr/nrv-inst#Driving
http://ns.inria.fr/nrv-inst#StateOfAffairs2 driving at 90km/h http://www.w3.org/2000/01/rdf-schema#label "driving at 90km/h"@en
[Gandon et al.]

PREDICT HOSPITALIZATION
▪ Predict hospitalization from
Physician’s records classification
▪ Augment records data with
Web knowledge graphs
▪ Study impact on prediction
[Gazzotti, Faron, Gandon et al. 2020]
Sexe Date Cause CISP2 ... History Observations
H 25/04/2012 vaccin-antitétanique A44 ... Appendicite EN CP - Bon état général - auscult
pulm libre; bdc rég sans souffle -
tympans ok-
Element Number
Patients
Consultations
Past medical history
Biometric data
Semiotics
Diagnosis
Row of prescribed drugs
Symptoms
Health care procedures
Additional examination
Paramedical prescription
Observations/notes
55 823
364 684
187 290
293 908
250 669
117 442
847 422
23 488
11 850
871 590
17 222
56 143
(1)
(2)
PRIMEGE

Image Metadata Score 
portrait
50350012455
C:Jocondejoconde0138m503501_d0012455-000_p.jpg
cheval:
0.999
Image Metadata Score 
figure (saint Eloi de Noyon, évêque, en pied, bénédiction,
vêtement liturgique, mitre, attribut, cheval, marteau, outil :
ferronnerie)
000SC022652
C:/Joconde/joconde0355/m079806_bsa0030101_p.jpg
cheval:
0.006
MonaLIA
▪ reason & query on RDF to build training sets.
▪ transfer learning & CNN classifiers on targeted
categories (topics, techniques, etc.)
▪ reason & query RDF of results to address
silence, noise and explain
350 000 images
of artworks
RDF metadata based
on external thesauri
Joconde database from French museums
(1)
(3)
[Bobasheva, Gandon, Precioso, 2021]
(2)

MonaLIA 2.0 Approach
• SPARQL+RDFS+SKOS on metadata to extract training and
test subsets of images
• create labeled training and test sets including the “narrower”
categories according to Garnier Thesaurus
• create “missing” links between some categories
• balance number of training images per class
• filter out certain categories and images
• Train Multi-Label Deep Learning classifier
• select state-of-the-art pre-trained CNN model
• adapt the model to multi-label classification
• fine-tune model on artwork images
• optimize model hyperparameters for best performance
• Apply trained model and extend metadata
• run all the images through the trained classifier
• record the prediction score as RDF triples
• SPARQL on extended metadata to search the database
(Maasai & Wimmics)

Detecting “noise”
By querying the extended metadata for the objects with low scores we
can detect the “noise” in the represented subject annotation
Image Metadata Score
figure (saint Eloi de Noyon, évêque, en pied, bénédiction, vêtement
liturgique, mitre, attribut, cheval, marteau, outil : ferronnerie)
000SC022652
C:/Joconde/joconde0355/m079806_bsa0030101_p.jpg
cheval: 0.006
figures bibliques (Vierge à l'Enfant, à mi-corps, assis, Enfant Jésus : nu,
livre);fond de paysage (colline, cours d'eau, barque, cavalier)
000PE027041
C:/Joconde/joconde0001/m503604_90ee1719_p.jpg
cheval: 0.009
scène (satirique : Bismarck Otto von : Gargantua, repas, cheval, boisson :
vin)
5002E006121
C:/Joconde/joconde0074/m500202_atpico-g70128_p.jpg
cheval: 0.011

Detecting “silence”
By querying the extended metadata for the object with high scores and
without object mentioned in annotation we can detect the “silence” in the
annotation
portrait
50350012455
C:Jocondejoconde0138m503501_d0012455-000_p.jpg
cheval: 0.999
scène historique (guerre de siège : Lawfeld, Louis XV, Saxe maréchal de,
bataille rangée)
000PE004371
C:Jocondejoconde0634m507704_79ee519_p.jpg
cheval: 0.999
figure (sainte Jeanne d'Arc, jeune fille, équestre passant, armure,
asque, épée)
M0301000355
C:Jocondejoconde0617m030106_007305_p.jpg
cheval: 0.997

Ranking of search results
Running the same query on the Extended Joconde database and sorting by
score gives a better result putting the image in the second place
représentation animalière (épagneul, debout)
M0341003743
chien: 0.994
scène (chasse : lévrier, lièvre)
M0810001165
chien: 0.993
représentation animalière (mise à mort, gros gibier : sanglier, chasse à
courre, chien)
00000105149
C:Jocondejoconde0107m505206_oa817_p.jpg
chien: 0.990

Hypermedia MAS
▪ Bridging Web architecture and Multi-Agent Systems architecture
▪ Hypermedia Communities of People and Autonomous Agents
▪ Define an architectural style for Hypermedia MAS
▪ Define declarative languages and mechanisms for specifying, enacting, and
regulating interactions among people and autonomous agents in
Hypermedia MAS
▪ Develop an open-source software infrastructure for Hypermedia MAS that
enables the deployment of hybrid communities on the Web
▪ Demonstrate the deployment of prototypical hybrid communities in two
application areas: (i) Industry 4.0 and (ii) tackling online disinformation.
http://hyperagents.gitlab.emse.fr/#

price
convert?
person
other sellers?
Web 1.0, 2.0, 3.0 …

Toward a Web of Programs
“We have the potential for every HTML document to be a
computer — and for it to be programmable. Because the thing
about a Turing complete computer is that … anything you can
imagine doing, you should be able to program.”
(Tim Berners-Lee, 2015)

Make the Web AI-friendly
content, links, metadata, etc.
data, knowledge, etc.
AI Web bots: chat bots, recommenders, facilitators, etc.
configuration, parameters, embeddings, services,
communication, etc.

deduce data

model, schemas, ontologies, ...
data data
15% progress

learn data
embeddings, parameters, configurations, …
data data
30% progress

sum intelligence

data data
45% progress

combine intelligence
data

60% progress

remotely combine
model, schemas, ontologies, …
embeddings, parameters, configurations,…

Web
75% progress

deeply combine
data, knowledge, model, schemas, ontologies, …
data, knowledge, embeddings, parameters, configurations,…

Web
90% progress

combining AIs on the Web
data, knowledge, model, schemas, ontologies, …
data, knowledge, embeddings, parameters, configurations,…

Web
100% progress

285
one Web … a unique space in every meanings:
data
persons documents
programs
metadata

Connected Animals, Animal-computer interaction (ACI)
Herdsourcing: monitoring collective animal behavior

IMAG_NE
a Web linking all forms of Intelligence

WIMMICS
1. user & interaction design
2. communities & social networks
3. linked data & semantic Web
4. reasoning & analyzing
epistemic hybrid communities
linked data
usages and introspection
contributions and traces

www
mmm
world wide web
massively multidisciplinary method

WIMMICS
Web-instrumented man-machine interactions, communities and semantics
Fabien Gandon - @fabien_gandon - http://fabien.info
he who controls metadata, controls the web
and through the world-wide web many things in our world.
Technical details: http://bit.ly/wimmics-papers
   

Web-Instrumented Man-Machine Interactions Research

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Web-Instrumented Man-Machine Interactions Research

Ähnlich wie Web-Instrumented Man-Machine Interactions Research (20)

Mehr von Fabien Gandon

Mehr von Fabien Gandon (17)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Web-Instrumented Man-Machine Interactions Research