Feedback-Driven Radiology Exam Report Retrieval with Semantics

1. Feedback-Driven Radiology Exam Report
Retrieval with Semantics
Sarasi Lalithsena
Amit Sheth
Kno.e.sis Center
Wright State University
Luis Tari
Steven Gustafson
GE Global Research
Ann von Reden
Benjamin Wilson
Brian Kolowitz
UPMC Enterprises
John Kalafut
GE Healthcare

2. 2
Motivation
Reason for exam
Secondary diagnoses related via
anatomical region or disease type
Follow up exam for an existing
condition
Specific hypothesis or inquiry
mentioned
To do an informed diagnosis radiologists might need:

3. 3
Problem
Reason for exam:
Foot pain
Access to wealth of data
EMR, Prior Radiology report
R1:……………………….
R2:……………………….
R3:History Diabetes, Smoker
……………………………
……………………………
Rn:...........................................
…..
R3:History Diabetes, Smoker
……

4. 4
Challenges
• Capture contextually relevant data
Patient X : Current Study
Reason for Exam: Foot Pain
…………………………………………………………………………………
R1 : Lower Extremity Venous Insufficient Doppler Ultra sound
R2 : History: Diabetes Former Smoker, History of Hypertension
RX : Generalized Abdominal Pain
……………………………………………………………………………………
……………………………………………………………………………………
Patient X : Prior Reports
Lower Extremity Venous
Insufficient Doppler Ultra
sound
History: Diabetes Former
Smoker, History of
Hypertension
Generalized Abdominal Pain
Foot partOf Lower
Extremity
Foot Pain associatedTo
Diabetes
Related by anatomical context
Related by disease context

5. 5
Challenges
• Capture contextually relevant data
Patient X : Current Study
Reason for Exam: Foot Pain
…………………………………………………………………………………
R1 : Lower Extremity Venous Insufficient Doppler Ultra sound
R2 : History: Diabetes Former Smoker, History of Hypertension
RX : Generalized Abdominal Pain
……………………………………………………………………………………
……………………………………………………………………………………
Patient X : Prior Reports
Lower Extremity Venous
Insufficient Doppler Ultra
sound
History: Diabetes Former
Smoker, History of
Hypertension
Generalized Abdominal Pain
Foot partOf Lower
Extremity
Foot Pain associatedTo
Diabetes
• Lexical Similarity
Measures fall behind when
there is no lexical similarity
• Knowledge-based
similarity measure only
uses taxonomical
relationships
Related by anatomical context
Related by disease context

6. 6
Challenges
• Personalize the relevancy based on radiologist's need
Patient X : Current Study
Reason for Exam: XR Chest, Rib Fracture
Abdomen X-RAY
Neuro SpecialistMusculoskeletal Specialist
More Relevant
MRI CHEST WALL More Relevant
XR Left Ribs including
CHEST
CT Thorax without
Contrast
All radiologists find these two records as relevant
Abdomen X-Ray was found relevant by Chest specialist and Musculoskeletal specialist
but less relevant by Neuro specialist, Generalists and Abdominal specialists

7. 7
Our Approach
Capture contextually relevant data
Knowledgebase and Semantic
Similarity
Personalize the relevancy
Explicit Relevant Feedback
Semantic
Vector Builder
Similarity
Calculator
……………………
R1 : Lower Extremity
R2 : History: Diabetes
……………..
………………
Prior Reports
……………………
R1
R2
Prior Report
Vectors
Reason for
exam vector
Feedback
Incorporator
R1 : Lower
Extremity
R2 : History:
Diabetes
…………

8. 8
Our Approach – Semantic Vector Creation
• Uses RadLex Ontology to capture semantic relationships with
NLP techniques
CT THORAX WITHOUT
CONTRAST
Reason for Exam: Chest pain
Reason for Exam: Foot pain
Lower Extremity Venous
Insufficient Doppler Ultra
sound

9. 9
Our Approach – Semantic Vector Creation
• Creates the vector of concepts using semantic similarity methods for
each relationship
𝑺𝒊𝒎 𝒄𝒊, 𝒄 𝒎 =
𝟎, 𝒊𝒇 𝑷𝒂𝒕𝒉𝑳𝒆𝒏𝒈𝒕𝒉 𝒄𝒊, 𝒄 𝒎 = 𝟎
𝟏
𝑷𝒂𝒕𝒉𝑳𝒆𝒏𝒈𝒕𝒉 𝒄𝒊, 𝒄 𝒎
, 𝑶𝒕𝒉𝒆𝒓𝒘𝒊𝒔𝒆
Lower Extremity Venous Insufficient
Doppler Ultra sound
1 0.5
bpi bpj
bpi HasPart bpj
bpi – Lower Extremity
bpj - Foot

10. 10
Our Approach – Semantic Vector Creation
• Handles multiple relation
𝒕𝒊 = 𝒎𝒂𝒙(𝑯𝒂𝒔𝑷𝒂𝒓𝒕𝒊 , 𝑰𝒔𝑨𝒊, … . 𝑪𝒂𝒖𝒔𝒆𝒔𝒊)
Lower Extremity Venous Insufficient
Doppler Ultra sound
1 0.5
1 0
1 0
0.2
0.02
HasPart
Isa
Causes
1 0.5 0.020.2

11. 11
Our Approach: Explicit Relevance Feedback
• Explicit relevance feedback is used in information retrieval to improve
the relevance of the results
• Useful when users have a general conception on what they are
looking for
• Adopted an existing algorithm “Rocchio Algorithm” to work with the
ratings provided by the users
*
!
* *
*
!
* *
*
!
!
* Relevant Record
! Non-Relevant Record
OO
MO
Original Query Vector
Modified Query Vector

12. 12
Our Approach: Explicit Relevance Feedback
Modified Rocchio query vector
𝒒 𝒎 = 𝒂𝒒 𝒐 + 𝒃
𝟏
𝑫 𝒓
𝒘𝒊
𝟓
𝒊=𝟏 𝒅𝒋𝒅 𝒋ε𝑫 𝒊
+ 𝒄
𝟏
𝑫 𝒏𝒓
𝒅 𝒌𝒅 𝒌ε𝑫 𝒏𝒓
𝑫 𝒓 = 𝑫 𝟏 𝑫 𝟐 𝑫 𝟑 𝑫 𝟒 𝑫 𝟓
qm - modified query vector
qo - original query vector
Dr - relevant patient records
Dnr - non relevant patient records
Di - patient records rated as i
a = 1 b = 0.6 c = 0.1

13. 13
Evaluation
• We use a radiology exam corpus consists of 6600+
anonymous exam reports to evaluate our approach
• Two evaluation strategies have been used,
– Domain expert based evaluation
– Intra-class and Inter-class similarity

14. 14
Domain Expert Evaluation – Part 1
• Evaluates the relevant patient records with 7 radiologists for 2 queries
• Measures the effectiveness of relevance retrieval via semantic vector
generation (without feedback) using Rprecision
• 𝑹𝑷𝒓𝒆𝒄@𝑹 =
𝒓
𝑹
r = number of relevant documents in top R
Q1 Q2
Rprec@2 Rprec@5 Rprec@8 Rprec@2 Rprec@5 Rprec@8
BOC 0.5 0.59 0.67 1 0.71 0.85
ISA 0.5 0.73 0.72 1 0.71 0.85
VSS 1 0.73 0.72 1 0.88 0.93
BOC: Only uses bag of concepts and no relationships
ISA: Only uses taxonomical relationships
VSS: Our approach which uses taxonomical relationships and HasPart relationships

15. 15
Domain Expert Evaluation – Part 2
• Evaluates the effectiveness for relevance feedback
• Re ranks the results with explicit feedback from domain experts
• R precision value increased from 0.72 to 0.78 for Q1
Study Description Before Feedback
BOC Ranking
After Feedback
BOC (R1) Ranking
XR LEFT RIBS INCLUDIND CHEST 1 1
ABDOMEN X-RAY 2 4
XR RIGHT FINGER(S) 2 3
CT THORAX WITHOUT CONTRAST 2 2
NUCLEAR MEDICINE PARATHYROID SCAN 5 9
BILITERAL VASCULAR ANKLE BRACHIAL INDEX 5 9
MR RIGHT KNEE 5 7
MR THORACIC SPINE 5 7
CT LEFT FOOT WITHOUT CONTRAST 5 5
MRI CHEST WALL WITHOUT CONTRAST 5 6

16. 16
Intra-class and Inter-class similarity
Hypothesis 1
All cases our approach would show a higher similarity score for Intra-
class than Inter-class.
Hypothesis 2
Differences will be increased with semantic enhancement for Intra-class
similarity and the differences will be decreased for Inter-class similarity

17. 17
Intra-class and Inter-class similarity
Experimental Set Up
● Group all patient records based on the exam code such as
CTCHEST, MAMSCRNDIG, DEXABOD and USABDCOMP
● Select 10 exam codes with the highest number of reports covering
about 2200+ reports
● Calculate the intra class and inter class similarity among these 10
exam codes (10 intra class similarities and 45 inter class similarities)

18. 18
Intra-class and inter-class similarity
• We show that intra-class similarities are always higher than inter-
class similarities
• Our approach was able to show that differences between intra-class
similarities are increased against the baseline (taxonomy approach)
• Our approach was able to show that differences between inter-class
similarities are decreased against the baseline (taxonomy approach)
Approach Avg Intra-class Sim
ISA 0.50
VSS 0.55
Approach Avg Inter-class Sim
ISA 0.178
VSS 0.162

19. 19
Conclusion
● We developed a technique that identifies relevant information from
prior radiology reports to help radiologists to improve their
interpretation
● We showed a semantic approach using Radlex ontology to find
relevant information from prior radiology reports was effective
● We showed the potential of using explicit relevant feedback to
personalize each radiologists need
● In future we plan to,
● Extend our corpora outside radiology reports
● Improve the coverage of the queries

20. Thank You!
Questions?
http://knoesis.wright.edu/researchers/sarasi
sarasi@knoesis.org
Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing
Wright State University, Dayton, Ohio, USA

21. 21
Hypothesis 1 Testing
All Intra-class similarities are higher than the inter-class similarities
A B C D E F G H I J
(A)CTCHEST 0.5625 0.1022 0.0902 0.0930 0.5193 0.1218 0.1183 0.1289 0.1383 0.1081
(B)MAMSCRNDIG 0.1022 0.7147 0.0841 0.2655 0.1070 0.0805 0.3519 0.2117 0.0862 0.3450
(C)DEXABOD 0.0902 0.0841 0.6031 0.0582 0.0914 0.1923 0.1410 0.1409 0.0766 0.1381
(D)USABDCOMP 0.0930 0.2655 0.0582 0.4496 0.0962 0.0959 0.1925 0.2059 0.1697 0.2101
(E)OSRCHESTCR 0.5193 0.1070 0.0914 0.0962 0.5380 0.1257 0.1231 0.1266 0.1361 0.1139
(F)LUMBARSP 0.1218 0.0805 0.1923 0.0959 0.1257 0.3482 0.1074 0.1021 0.1123 0.1003
(G)MAMDIAGDIG 0.1183 0.3519 0.1410 0.1925 0.1231 0.1074 0.5954 0.2742 0.1364 0.2916
(H)USSOFTISSU 0.1289 0.2117 0.1409 0.2059 0.1266 0.1021 0.2742 0.5782 0.1364 0,2916
(I)CTABDPEL 0.1383 0.0862 0.0766 0.1697 0.1361 0.1123 0.1364 0.1364 0.4995 0.1474
(J)USBREASTLT 0.1081 0.3450 0.1381 0.2101 0.1139 0.1003 0.5165 0.2916 0.1474 0.5623

22. 22
Hypothesis 2 Testing
Approach Avg Intra-class Sim
ISA 0.50
OUR 0.55
Differences are increased
for intra-class similarity

23. 23
Hypothesis 2 Testing
Approach Avg Inter-class Sim
ISA 0.178
OUR 0.162
Differences are decreased
for inter-class similarity