1. Ramesh Raskar discusses using AI, sensors, and interactive technology to augment surgeons by creating a "Waze for surgeons" that allows sharing tips, rapid training, and detecting anomalies. 2. He proposes capturing millions of surgery videos to analyze with AI and create a library of complications. 3. The goal is to guide surgeons with augmented reality, haptics, and screens to improve precision health for individual patients and population health across surgical patients.

1. Camera Culture
Ramesh Raskar
Ramesh Raskar
Assoc. Professor
MIT
Augmenting Surgeons
Patient to Population Scale
AI and Sensors

4. Capture:
Millions of Surgery Videos
Library of Complications
Analyze:
AI (CV/ML/NLP)
Tips, Anomalies
Interact:
Guide with AR/
Haptics/ Screens

5. Head-mounted Display Projectors
© Andrei State
Augmented Surgeons: Beyond X-ray vision
Multi-Mode AR
Raskar PhD Thesis ~2000

6. Raskar | MIT Media Lab
Augmented Reality (AR)
World Input Process Output World
Sense beyond
human ability Abstract
Present within
human ability

7. Raskar | MIT Media Lab
Augmented Reality (AR)
World Input Process Output World
Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
Overlay
Interaction
Resolution,
FoV, FPS,
Optics
Bounds of
Physics + Perception

8. Raskar | MIT Media Lab
South Park - Jurassic Park

9. Raskar | MIT Media Lab
Realism
Functional
Photoreal
Physical
Non-visual
Ferwerda 2003

10. Raskar | MIT Media Lab
Realism
Functional
Photoreal
Physical
Non-visual
Archaic neurosc +
perception
Few open datasets

11. Raskar | MIT Media Lab
Multi-ModeMulti-Format
IoT
Perception
Ergo,
Hygiene
Gestural Language

12. Raskar | MIT Media Lab
Augmented Reality (AR)
World Input Process Output World
Camera,
Location
Sensors
Identify,
Track,
Abstract
Display,
3D Overlay
Interaction
MultiModal MultiFormat
IoT
Training
DataSets
Perception
Ergo, Hygiene
Ethics
Gestural Language
EyeGlasses
Authoring
IoT

13. Raskar | MIT Media Lab
hand-held
display
spatial
see-through
display
head-
mounted
display
retinal
display
Head-attached Body-attached spatial
projector
Spatial Augmented Reality
1998
Multi-Mode AR
Real
Object

14. Raskar | MIT Media Lab
Virtual Reflectance Virtual Illumination
InteractionVirtual Motion
ShaderLamps
Raskar PhD Thesis 1997 - 2000

15. Machines will not replace talented hands of surgeons
Worldwide scarcity of Surgeons
1. Augmenting Surgeons
‘Waze’
2. Precision Health + Population Health for Surgery
‘Anatome’, Omics for Anatomy
Patient to Population Scale
Raskar PhD Thesis 2002

16. Waze for Surgeons
Decision making, Share Tips, Training, Find Anomalies

18. Capture:
Millions of Surgery Videos
Library of Complications
Analyze:
AI (CV/ML/NLP)
Tips, Anomalies
Interact:
Guide with AR/
Haptics/ Screens

19. Capture Analyze Interact
Improve
Video, Haptics
Speech Interface
TransformDisrupt
Handle co-morbiditiesLibrary of
Complications
Rapid Training,
Create Olympians
Sharing tips and tricks
On site
diagnostic tests
Avoid Waiting
Manage teams
Find anomalies early
Display and Haptics
Precision+PopulationHealth
Waze
Post-operative Predictions

20. Anatome
Lee Sanders, Raskar 2018
Bard et al 2006
Hashimoto, Rus et al 2016
Gillies, 2014
structure,
function,
development,
evolution,
network

21. Capture Analyze Interact
Improve
Video, Haptics
Speech Interface
TransformDisrupt
Handle co-morbiditiesLibrary of
Complications
Rapid Training,
Create Olympians
Sharing tips and tricks
Novel Surgeries without
special training
Cellular Level,
Multispectral
On site
diagnostic tests
Avoid Waiting
Manage teams
Find anomalies early
Display and Haptics
Remote /
Distant Capture
‘Anatome’: Anatomic
structure Library
Precision Health:
Guided Options
Population Health across
surgical patientsPrecision+PopulationHealth
Waze + Anatome
Post-operative Predictions

22. Challenges
•Capture:
•How to get labelled millions of surgery videos
•Analyze:
•Bias in data, close viewpoint of procedures
•Interact:
•Avoid cartoonish VR, Will UX get in the way

23. Capture Analyze Interact
ImproveTransformDisrupt
InstrumentScaleWorldScale
Machine Learning
Computer Vision
IoT
Motion Tracking
Raskar Team’s Prior Work
Auto-ML
Camera to see
around corners
Brush Endoscope
Cell-resolution Imaging
Tricorder Diagnostics
8D Telehealth
Haptics
Retinal Predictions
Book on AR
Leader @Oculus VR
90 Patents
250 Pubs, Many startups

24. Acknowledgements
• Camera Culture Group, MIT Media Lab
• Lee Sanders, Carla Pugh, Raj Gupta, Daniel Kraft
• John Werner, Marco Jacobs, Mark Billinghurst, Sameer Rawal, Tim Smith,
Alvaro Cassinelli, Benjamin Lok, Rob Lindenman, Brett Jones, Mk Haley,
Chris Sanders
• MERL
• Amit Agrawal, Ashok Veeraraghavan, Ankit Mohan, Jack Tumblin, Jeroen van Baar, Paul
Beardsley, Remo Ziegler, Thomas Willwacher, Srinivas Rao, Cliff Forlines, Paul Dietz, Joe Marks,
Darren Leigh
• UNC Chapel Hill, Office of the Future group
• Greg Welch, Kok-lim Low, Deepak B’padhyay, Aditi Majumder, Michael Brown, Ruigang Yang
• Henry Fuchs, Herman Towles
• Wei-chao Chen

26. Machines will not replace talented hands of surgeons
Worldwide scarcity of Surgeons
AI, Sensors, Interactive Tech
1. Augmenting Surgeons
‘Waze’ for Surgeons
Sharing tips, Rapid education, Anomalies
Tech to get you in and out of OR
2. Precision Health and Population Health for Surgery
‘Anatome’: structure, function, development, evolution
Predict complications, handle co-morbidities
Novel surgeries without special training
AR for IA
Patient to Population Scale for Surgery
Ramesh Raskar, MIT Media Lab