May 2021 snapshot of some of the Research and Collaborations in dHealth/personalized health, public health, epidemiology, biomedicine at the AI Institute of the University of South Carolina [AIISC]
1. Healthcare Research @ AI Institute:
dHealth, public health, epidemiology, biomedicine
Overview Presentation to the MUSC’s AI Hub and others
May 2021
Amit Sheth, Founding Director http://aiisc.ai
3. “
3
AIISC in core AI areas, and
interdisciplinary AI/AI applications
>> 25 researchers
including 4 faculty
(6 in Fall 2021), 2-3
postdocs, ~20 PhD
students, >10
MS/BS and several
interns/associates
5. D-Health, health informatics, public health, epidemiology –
sample collaborations (Pending & PLANNED Submissions only)
•College of Medicine/Prisma/Prisma-Upstate
•Mental health [M Natarajan], Addiction [A. Litwin], Asthma [R. Dawson], Diabetes & Obesity
[L. Knight], Hypertension - Diet & Nutrition [S. Donevant], Neutropenia [S. Craemer]
•College of Pharmacy: EOCRC [P. Backhaults, L. Hofseth, et al]
•College of Nursing: COVID-19 mobile app [R. Hughes, S. Donevant], mental health
chatbot [R. Hughes, S. Donevant, P. Raynor]
•Arnold School of Public Health: Mental Health [S. Qiao], Healthcare Big Data -
Education & Training [X. Li, et. al.]
•USCAND, Inst of Mind & Brain: Neuroscience [R. Desai] and neurodevelopmental
diseases [J. Bradshaw, J. Roberts]
•CEC- IIT, BME, CSE: Health IT/Smart Health [E. Regan], UI/UX [D. Wu]; Health
mApp [N. Boltin], (several in CSE).
6. Projects
➢ KHealth: Semantic Multisensory Mobile Approach to Personalized Asthma Care (NICHD)
➢ mHealth to Improve Carbohydrate Counting Accuracy in Pediatric Type 1 Diabetes
➢ Improving mental health of COVID-19 patients with an Artificial Intelligence-based chatbot
➢ Personalized Virtual Health Assistant Enabled by Knowledge-infused Reinforcement Learning for Adaptive
Mental Health Self-management
➢ Characterizing and supporting help seekers on social media using expert-in-the-loop learning
➢ Modeling Social Behavior for Healthcare Utilization in Depression (NIMH)
➢ eDarkTrends : Monitoring Cryptomarkets to Identify Emerging Trends of Illicit Synthetic Opioids Use
(NIDA)
➢ Trending: Social media analysis to monitor cannabis and synthetic cannabinoid use (NIDA)
➢ Innovative NIDA National Early Warning System Network (iN3) (NIDA)
➢ PREDOSE: PREscription Drug abuse Online Surveillance and Epidemiology (NIDA)
➢ Community-Driven Data Engineering for Substance Abuse Prevention in the Rural Midwest (NSF)
➢ Discrepancies in Diagnosis and Treatment of Cardiovascular Disease Based on Sex and Gender to Improve
Women’s Health (NHLBI)
➢ Early Onset of Colorectal Cancer
➢ Digestive Inflation Index
➢ more ...
7. Types of Healthcare Data
◎ EMR
◎ Social Media (Reddit,
Twitter,Web Forums)
◎ Conversations: Patient-
Clinician, Virtual Health
Assistant-Patient
◎ Patient Generated:
Wearable/sensor data,
mApp data
◎ Images: food, fMRI
AI Techniques and Technologies
◎ Knowledge Graphs/Ontologies
(contextualization, personalization,
abstraction)
◎ NLP/NLU
◎ Machine Learning/Deep Learning
(RL, GAN, CNN, LSTM,....)
◎ Conversational AI, Q/A
◎ mApp, Virtual Health Assistants
(Chatbots)
◎ Health sensors/IoTs/mobile
devices
8. Medical Conditions/Healthcare Challenges addressed
◎ Asthma
◎ Mental Health
◎ Addiction
◎ COVID-19
◎ Cardiovascular Disease
◎ Type 1 Diabetes in Children
◎ Adult Diabetes - Hypertension
◎ Neutropenia
◎ Sleep Disorders
◎ Gender and Race Disparity
◎ Demographics
◎ SDOH
◎ Drug Design
Partners:
Weill Cornell, UCSF Medical,
Prisma-Health, Addiction
Research Center, UofSC
Medical/Pharma/Public
Health/Nursing; Wright State
Physicians, ….
9. Unique Value Propositions and Strengths [for Health Apps]
◎ Development of Knowledge Graphs
◎ Knowledge-infused (Deep) Learning and Knowledge-
infused NLP: Explainable AI
◎ Conversational AI/collaborative agents
◎ Augmented Personalized Health
10. 10
Health Knowledge Graph
Drug Abuse Ontology
Lokala U, Daniulaityte R, Lamy F, Gaur M, Thirunarayan K, Kursuncu U, Sheth. A. (2020).
DAO: An ontology for substance use epidemiology on social media and dark web. JMIR.
https://doi.org/10.2196/preprints.24938
https://scholarcommons.sc.edu/aii_fac_pub/356/ [Shah and Sheth US patent 2015]
11. Knowledge-Infused
Learning with
Semantic,
Cognitive,
Perceptual
Computing
Framework
11
Overarching Theory
Knowledge
Domain (Ontology)
Personalized KG
Multisensory
Sensing &
Multimodal
Data Interactions
Images
Text Speech Videos
IoTs
Natural Language
Processing,
Machine with
Deep Learning
AUGMENTED PERSONALIZED
HEALTH (APH)
Modeling broader disease context, and
personalized user behavior
Reasoning & decision-
making framework
Minimize data overload, assist in making
choices, appraisal, recommendations
TEDx talk: Augmented Health with Personalized Data and AI
12. 12
Contextualization and
Personalization
kBOT initiates greeting
conversation.
Understands the patient’s health
condition (allergic reaction to high
ragweed pollen level) via the
personalized patient’s knowledge
graph generated from EMR,
PGHD, and prior interactions with
the kBot.
Generates predictions or
recommended course of actions.
Inference based on patient’s
historical records and
background health knowledge
graph containing contextualized
(domain-specific) knowledge.
Figure: Example kBot conversation which
utilizes background health knowledge graph
and patient’s knowledge graph to infer and
generate recommendation to patients.
★ Conversing only information relevant
to the patient
Context enabled by relevant
healthcare knowledge including
clinical protocols.
13. Why Knowledge
Infused Learning
(K-IL)?
By changing the inputs, it can
enrich the representation (E.g.
Radicalization on Social Media)
By changing parameters, we can
control the learned
patterns/correlations to adhere to
the knowledge.
Deep Infusion would allow us finer
grained control over learned
patterns to ensure adherence to
knowledge at every step of the
hierarchy
Explanations easy to derive from
the KG used 13
Contextual Modeling to
analyze Radicalization on
Social Media
(Hate)
16. What? Comprehensive, Customizable, Adaptive
● App/chatbot for individuals
○ Cohort 1: College of Nursing (testing, research)
○ Cohort 2: Students, Staff, Faculty
● Dashboards for different uses: general administration, health services, research
○ Support for custom protocol
Gamecock look-n-feel, easy to use and engaging, customizable, secure, privacy
disclosure/management, HIPAA compliance, accessibility (ADA compliance), scalable, well
tested, extensible
17. Why? Proactively keep Campus healthy
● Informing and Educating the Community, Regularly Check Health Status:
○ Stay informed and educated, make better individual decisions, feel safer, reduce
adverse outcomes
○ In case of concern, connect with Student Health Services, manage isolation protocols
● Providing Campus-wide View: tools needed to make campus-wide decisions
○ Insurance against possible adverse outcome from low-risk approach (CDC’s
Considerations for Institutes of Higher Education (May 21), proactive
implementation of protocols
● Research: COVID-19, Mental Health
18. How?
● Comprehensive campus-wide involvement and coordination: requirements,
development, testing, operations
○ College of Nursing: lead evaluation
○ Student Health Services
○ School of Public Health
○ Division of Information Technology
○ CEC; of course, the AI Institute
● Development team with extensive experience
● Much more functional, forward looking, and cheaper than vendors
20. Demo: Health-e Gamecock COVID-19 App (WebApp, IOS, Android)
Note: development is now complete, app is evaluated and we plan to use it for a major study.
24. 24
Use Case: kHealth Asthma
Many Sources of Highly Diverse Data
(& collection methods: Active + Passive):
Up to 1852 data points/ patient /day
http://bit.ly/kHealth-Asthma
kBot with screen
interface for conversation
Images
Text
Speech
★ Episodic to Continuous Monitoring
★ Clinician-centric to Patient-centric
★ Clinician controlled to Patient-empowered
★ Disease Focused to Wellness-focused
★ Sparse data to Multimodal Big Data
*(Asthma-Obesity)
25. Self Monitoring with kHealthDash:
Knowledge enabled personalized DASHboard for Asthma Management
Video link - https://youtu.be/yUgXCPwc55M
26. Digital Phenotype Score vs Asthma Control Test Score
Digital Phenotype Score = Symptom Score + Rescue Score + Activity Score + Awakening Score
27. 27
Utkarshani Jaimini, Krishnaprasad
Thirunarayan, Maninder Kalra, Revathy
Venkataramanan, Dipesh Kadariya, Amit
Sheth, “How Is My Child’s Asthma?”
Digital Phenotype and Actionable Insights
for Pediatric Asthma”, JMIR Pediatr Parent
2018;1(2):e11988, DOI: 10.2196/11988.
28. Self Appraisal with Digital Phenotype Score
Jaimini U, Thirunarayan K, Kalra M, Venkataraman R, Kadariya D, Sheth A, “How Is My Child’s Asthma?” Digital Phenotype and Actionable Insights for Pediatric Asthma, JMIR Pediatric Parent 2018;1(2):e11988
https://medium.com/leoilab/digital-phenotyping-turning-our-smartphones-inward-141a75b2f2a3
● Digital Phenotype Score (DPS) is defined as the score
to quantify the digital phenotypes collected from the
social media, smartphones, wearables, and sensors
streams.
● DPS acts as a cumulative measure for the abstraction
of knowledge and information from the raw digital
phenotypic data.
● The integration of the DPS can enable personalized
interventions in real time which are directly responsive
to the healthcare need of a patient.
29. Using Knowledge Graphs to construct a contextualized and personalized profile for each patient
that can drive insights and personalized care strategies
30. ● Published in ISWC 2018 Contextualized Knowledge Graph Workshop, 2018. Amelie
Gyrard, Manas Gaur, Saeedeh Shekarpour, Krishnaprasad Thirunarayan, Amit Sheth
2018, ISWC.
● Sheth, A., Jaimini, U., & Yip, H. Y. (2018). How Will the Internet of Things Enable Augmented
Personalized Health? IEEE Intelligent Systems, 33(1), 89–97.
https://doi.org/10.1109/MIS.2018.012001556
31. Determining Personalized Asthma Triggers: Seasonal Dependency
Venkataramanan R, Thirunarayan K, Jaimini U, Kadariya D, Yip HY, Kalra M, Sheth A. Determination of Personalized Asthma Triggers From Multimodal Sensing and a Mobile App: Observational Study. JMIR Pediatr Parent 2019;2(1):e14300. doi: 10.2196/14300
PMID: 31518318 PMCID: 6716491
33. Evidence based Path to Personalization
Patient-A was monitored for 13 weeks encompassing winter to spring 2018. Type: Severe, low medication compliance.
Absence of Pollen
First 6 weeks
Presence of Pollen
Rest of the 7 weeks
Pre (observe)
4 weeks
Post (validate)
2 weeks
Pre
4 weeks
Post
3 weeks
Pollen 0 Pollen 0 days Pollen 17 days Pollen 3 days
PM2.5 20 days PM2.5 5 days PM2.5 14 days PM2.5 2 days
Ozone 1 day Ozone 0 Ozone 0 Ozone 1 day
Asthma
Episodes*
21 days Asthma
Episodes
5 days Asthma
Episodes
17 days Asthma
Episodes
3 days
● Absence of Pollen - PM2.5 is the trigger
● Presence of Pollen - Pollen and PM2.5. Severe symptoms occurred in this period. Presence of both PM2.5 and
Pollen increased the intensity of asthma episodes.
Venkataramanan R, Thirunarayan K, Jaimini U, Kadariya D, Yip HY, Kalra M, Sheth A. Determination of Personalized Asthma Triggers From Multimodal Sensing and a Mobile App: Observational Study. JMIR Pediatr Parent 2019;2(1):e14300. doi: 10.2196/14300
PMID: 31518318 PMCID: 6716491
36. 36
Q1: Do you feel restless in sleep?
A1: Two-three times a week
Q2: Do you wake early? How often does it happen?
A2: Too early, happens two times a week.
Q3: Do you persistently feel sad?
A3: Yes. A sense of loneliness
Q4: How often you stay alone?
A4: I am divorced.
Q5: Have you been diagnosed with PTSD?
A5: No
Q6: Have you seen an MHP for your anxiety disorder?
A6: hmm recently.
Participant was asked do they feel restless in sleep,
then participant said hmm,two three times week
Participant was asked, do they persistently feel sad,
participant said divorce loneliness
Participant was asked, have they been diagnosed with
PTSD, participant said hmm recently.
Diagnostic Interview Summaries
Contextualized Diagnosis of
patient conversations
Correct diagnosis: Post Traumatic Stress
Disorder
42. Interest
We are interested in
Matching Support Seekers
-SSs (left) with Support
Providers - SPs (right)
Current State
Currently, moderators
(center) do this matching
42
Proposal
Our AI system will replace/assist the moderators that use
medical knowledge, information about the user, extracted
from the posts to perform this matching
48. Knowledge Graph for better Information Extraction: Application
in Epidemiology
48
Cameron, Delroy, Gary A. Smith, Raminta Daniulaityte, Amit P. Sheth, Drashti Dave, Lu Chen, Gaurish Anand, Robert Carlson, Kera Z. Watkins, and Russel Falck.
"PREDOSE: a semantic web platform for drug abuse epidemiology using social media." Journal of biomedical informatics 46, no. 6 (2013): 985-997.
50. K-IL: Shallow Infusion with DAO in DL model to detect trends in
cryptomarkets
Table: Sample properties derived from cryptomarket with DAO
51. Motivation
● The opioid epidemic entrenched in
Ohio and the Midwest of the US.
● The prevalence of opioid and its
impact on the well-being of
individuals and the society in Ohio.
○ Mental Health & Suicide Risk
Questions
1. How can we use social media to measure
mental health impact of opioid
prevalence?
1. Are there association between opioid and
mental health/suicide risk based on social
media data?
Approach
Monitoring the prevalence of opioid and its impact on mental health and suicide in Ohio,
utilizing a scalable knowledge and data driven BIGDATA (BD) approach via social media.
BD Spoke: Opioid and Substance Use in Ohio
52. Score
Calculation
Opioid
Mental Health
Depression
Addiction
Suicide Risk
Ideation, Behavior
Attempt
Correlations
● Sheth, Amit, and Pavan Kapanipathi. "Semantic filtering for social data." IEEE Internet Computing 20, no. 4 (2016): 74-78.
● Hussein S. Al-Olimat, Krishnaprasad Thirunarayan, Valerie Shalin, and Amit Sheth. 2018. Location Name Extraction from Targeted Text Streams using Gazetteer-based Statistical
Language Models. In Proceedings of the 27th International Conference on Computational Linguistics (COLING2018), pages 1986-1997. Association for Computational Linguistics
● Gaur, M., Kursuncu, U., Alambo, A., Sheth, A., Daniulaityte, R., Thirunarayan, K., & Pathak, J. (2018, October). " Let Me Tell You About Your Mental Health!" Contextualized
Classification of Reddit Posts to DSM-5 for Web-based Intervention. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (pp. 753-
762).
● Gaur, M., Alambo, A., Sain, J. P., Kursuncu, U., Thirunarayan, K., Kavuluru, R., ... & Pathak, J. (2019, May). Knowledge-aware assessment of severity of suicide risk for early
intervention. In The World Wide Web Conference (pp. 514-525).
● Yazdavar, A. H., Al-Olimat, H. S., Ebrahimi, M., Bajaj, G., Banerjee, T., Thirunarayan, K., ... & Sheth, A. (2017, July). Semi-supervised approach to monitoring clinical depressive
symptoms in social media. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017 (pp. 1191-1198).
● Daniulaityte, R., Nahhas, R. W., Wijeratne, S., Carlson, R. G., Lamy, F. R., Martins, S. S., ... & Sheth, A. (2015). “Time for dabs”: Analyzing Twitter data on marijuana concentrates across
News
Articles
Twitter
Data
Domain
Knowledg
e
Content
Enrichment
DAO
DSM-5
Location Extraction
Keyphrase Extraction
Age-based
Clustering
Semantic Filtering
Entity
Extraction
NLM Training
f(.)
Knowledge Infused
Natural Language
Processing (Ki-NLP)
Semantic
Mapping
Semantic
Proximity
Topic Model
Language Model
DAO
DSM-5
Dashboard
Visualizations
(Online)
Offline
Analysis
&
Visualizations
BD Spoke: Opioid and Substance Use in Ohio
53. ● Substance use addictive disorder linked to
opioid with higher correlation.
● Gender dysphoria, Dissociative and OCD
disorders are correlating moderately.
Opioid Prevalence in Ohio vs. Mental Health & Suicide
● Suicide ideation (initial stage) with highest correlation.
● Mild severity level of suicide risk linked to higher
correlation.
● Weak correlation for suicide indication (before initial)
p
N counties
p
N counties
54. with a Social Quality Index
Insights from semantic analysis of Social Media Big Data
Psychidemic: Measuring the Spatio-Temporal
Psychological Impact of Novel Coronavirus
with a Social Quality Index
Insights from semantic analysis of Social Media Big Data
○ Mental Health: Depression, Anxiety
○ Addiction: Substance use/abuse
○ Gender-based/Domestic Violence
○ Mental Health: Depression, Anxiety
○ Addiction: Substance use/abuse
○ Gender-based/Domestic Violence
55. Capability Demonstration 2
● Spatio-temporal analysis of big data (1billion tweets,
700,000 articles)
● Use of domain knowledge graphs (mental health,
addiction)
● Complex language understanding (slang, specialized
domain terms)
○ Can do people/demographic, network, sentiment, emotion, intent
analysis
● Scenario: Understand the implication of policy
choices (e.g., school/business closing) and real-
world events during COVID-19
55
56. A calculated Social Quality Index (SQI)
aggregates mental health components
(Depression, Anxiety), Addiction and
Substance Use Disorders.
Social Quality Index (SQI)
vecteezy.com
● Change in SQI informs comparisons between
states.
● Raw transformed SQI into relative state rankings
changing over time.
57. e.g., IN, NH, OH,
OR, WA, WY
are worsening.
Results: Relative State Rankings Reveal Patterns
SQI Ranking April 4 - 10
SQI Ranking March 14 - 20 SQI Ranking March 21 - 27
SQI Ranking March 23-April 3
Darker: Better
Social Quality
58. Results: Cluster --Improving SQI Ranking
SQI bad SQI better SQI better
SQI better
Frequency
Depression: 125037
Addiction: 92897
Anxiety: 81891
Total: 299825
Frequency
Depression: 113830
Addiction: 81810
Anxiety: 74080
Total: 269720
Frequency
Depression: 81463
Addiction: 60166
Anxiety: 45998
Total: 187627
Frequency
Depression: 59088
Addiction: 49086
Anxiety: 46887
Total: 155061
IL, NY, MD,
AZ, NM, MA.
March 14-20 March 21-27 March 28-April 3
April 4-10
59. External Events
● Stay at home order
● Extension non-essential closure
● Closing parks
● State of emergency
● School Closure
● Mental Health Alarm
● Extension of small business closure
● Bill payment deadline extended
● 41k new job openings
● Child-care assistance for essential
workers
● Spike in number of
cases
● Stay at home order
extended
● Extension School
Closure
● State of emergency
Extension
● Unemployment
Increase(>800%)
● Tax deadline extended
● SNAP Benefits
● Death Benefits
● Domestic Violence Alarm
● Spike in number of cases
● Stay at home order extended
● Extension School Closure
● State of emergency Extension
● Closure barber shops and related
businesses
● Number of deaths cross 50000
● Unemployment Increase(>2.5k%)
● Tax deadline extended
● Phases reopening
● Limited indoor seating or gathering
● CARES Act
Change
in
SQI
relative
to
first
week
64. Knowledge-infused Reinforcement Learning
● The input to the agent is sequential through many steps, it gets an input and a reward at every step and
learns the right output gradually through reinforcement.
66. NOURICH
A system to monitor diet,
recommend meals and promote
healthy eating habits:
Current application: Type 1 diabetes
in Children
Contact: Revathy Venkataramanan
Acknowledgement: Thanks to my collaborators Hong Yung Yip and Thilini Wijayasriwartane for the slides
NOURICH
Know What you Eat
67. Overweight Obesity Hospitalization
Prevent
overweight
moving to
obesity
Prevent obesity
leading to
hospitalization
Bridging the gap
“Focusing on reducing excess and impulsive calorie
consumption and making an informed decision about food
choices and physical activity can help one attain a healthier
weight and minimize the risk of chronic illness”
The Dietary Guidelines of Americans 2010
68. ● Real-time food recognition
● Tensorflow Mobile net model
20
Food Categories
700
Images/Category
Image Data Source
Image Source: https://commons.wikimedia.org/wiki/File:Google_%22G%22_Logo.svg, https://freepik.com
Nutrition Management System
NOURICH
GOAL: A system to monitor
diet, recommend meals and
promote healthy eating habits.
69. Architecture for application to Type 1 Diabetes in Children
Data sources
- User specific (food allergies,
comorbidities, lab reports
including genetic profiles and
etc)
Personalized
Knowledge Base
- Meal name
- Nutrition
- Ingredients
- Cooking style
Data collected
DATA STORE
Data collected are stored along with
domain knowledge
Image
Voice
Text
Inputs
Processing engine
Image, voice,text to
keyword
DASHBOARD
- Carbohydrate count
70. Trained on
Food Images
Cheesecake
NOURICH
Bitmap
Conversion
ByteArray
Conversion
3 frames per
second
Python
Script
Data Cleaning
1) Removing png
2) Removing non jpeg
Shell
Script
Image
Annotation
Training
Model: Mobile-net
model
Accuracy: 83%
Graph
protobuf file
Data preprocessing and training layer
Crawled Images
Conversion to
TFLite
Image Recognition layer
Recognized image is displayed to the user
APPLICATION ARCHITECTURE
User
Display nutrition
info for the food
Food Logs
Sources: 1) User Icon by Gregor Cresnar from the Noun Project, 2)Food Images from Google Images, 3)TensorflowLite,Nutritionix,Bash, Instagram, Google - logos are from original vendors
71. Matching Support Seeker (SS) with Support Provider (SP)
Online: Current application - matching SS and SP on mental
health related subReddit
https://scholarcommons.sc.edu/aii_fac_pub/516/
73. 73
Modeling Exogenous Information into Epidemiological Models
(Exo-SIR)
Curve of Exogenous Information in Tamil Nadu
Infected Curve (yellow) shift left because of
Exogenous Information (Exo-SIR)
Architecture of Exo-SIR model
Simple SIR
Model
Infected
Curve for
Tamil Nadu
State
● The curve representing time to infection shifts left (28%
early) when we introduce exogenous events such as large
gatherings (eg. Tablighi Jamaat religious gathering) or labor
migration due to lockdown.
● Evaluated/validated for the impact of exogenous events on
three states in India (Rajasthan, Tamil Nadu, Kerala).
(Accepted at AI for COVID track in ACM KDD 2020)
74. There is a lot more:
http://aiisc.ai, http://wiki.aiisc.ai
https://scholarcommons.sc.edu/aii_fac_pub/
Many thanks to our sponsors, esp. ~10 NIH
grants (four R01s, three R21s, R56, etc) from
NIMH, NIDA, NICHD, and other institutes.