Design to accommodate “intelligent adaptive experiments” with future-proof hardware for deep learning-enabled imaging and neuroscience. In other words, how to design future-proof measurement systems that are both easy to setup and are scalable for more advanced measurement paradigms of the future. And how you would like to think of structuring your data acquisition to be used efficiently with deep learning in neuroscience. Alternative download link: https://www.dropbox.com/s/j5r8vifvh6e7bfp/animal_instrumentation.pdf?dl=0

1. Instrumentation
for in vivo intravital
microscopy
Design to accommodate
“intelligent adaptive
experiments” with future-proof
hardware for deep learning-
enabled imaging and neuroscience
Petteri Teikari, PhD
Singapore Eye Research Institute (SERI)
Visual Neurosciences group
http://petteri-teikari.com/
Version “Mon 5 November 2018“
Figuresfrom
https://doi.org/10.1186/1752-0509-2-74
https://doi.org/10.3389/fphys.2015.00147
https://www.nikonsmallworld.com/people/wim-va
n-egmond

2. BasicsofanI/Osystem

3. DAQDataAcQuisition System
System that converts your analog signal (e.g. EEG, ECG,
temperature, blood pressure, etc.) to a digital signal
stored ona computer
NationalInstruments
DAQs
http://www.ni.com/data-acqui
sition/
Worksbestwiththe LabVIEW
(developedbyNational
Instruments)virtual
instrumentation software
LabJackDAQs
https://labjack.com/products/c
omparison-table
Worksforexamplewith
PsychoPy(Python)ifyouare
running behavioral
experiments.
BitScope and Raspberry Pi
http://www.bitscope.com/blog/DI/?p=DI25A
BitScopecan capturemultipleanalog and digital
signalsatveryhigh samplerates(up to 40MSpsin
somecases)withoutloading theRaspberryPi CPU
or requiring areal-timeoperating systemfor low
jittersampling.

4. CommercialSystemsforBiosignalinstrumentation

5. Input–sensor /
Output– actuator,trigger,stimulus,etc.
Non-intelligent“datalogging”
Simplyjustlogthesensorreadingstodisk
HOBO Pendant UA-002-64
Temperature/LightDataLogger
If youare onlyinterested in
temperature andlightof your
animalhousingwith noneedto
combine thesemeasurements
with any other analysis,this
approachmightbeenough

6. Input–sensor /
Output– actuator,trigger,stimulus,etc.
A“bitmoreIntelligent”data logging
Youhavemultiple heterogeneous sensors and you want common timestamp
Biopac wirelessECGsystemformice/ rats
https://www.biopac.com/product/epoch-
wireless-ecg/
YourmaininterestistheECG,
but youmight want tomake sure
thatthe environmental factorsdo
notconfoundyourmeasurements

7. Input–sensor /
Output– actuator,trigger,stimulus,etc.
Addsomefeedbackto keepenvironmentstable
Implement a PIDcontroller with Arduino / Raspberry Pi

8. ‘Circadiansidestep‘
forquantifyingthelightenvironment

9. Input–sensor /
Output– actuator,trigger,stimulus,etc.
inLight/ Circadian
studiesyou wantto
makesurethat
luminance
distribution,
intensityand
spectralcontent
arethedesired
http://dx.doi.org/10.12688/wellcomeopenres.9892.2
COMPASS: Continuous Open Mouse Phenotyping of Activity and Sleep Status
“Finally the authors would like to thank the open-source communities
connected to Arduino, Processing, Python/PyData stack and Blender for the
toolsusedto illustratethemethodsinthispaper.”

10. Input–sensor /
Output– actuator,trigger,stimulus,etc.
inLight/ Circadian
studiesyou wantto
makesurethat
luminance
distribution,
intensityand
spectralcontent
arethedesired TheBee-Eye-LuminanceDistribution
MeasurementsOptiLight-Mathematical
OptimizationsforHumanCentricLighting
ThijsKruisselbrink,RajendraDangol&Alexander
Rosemann,BuildingLightingGroup,DepartmentofBuilt
Environment
RaspberryPi-based
lowcostsolution with
afisheyelens

11. Input–sensor /
Output– actuator,trigger,stimulus,etc.
inLight/ Circadian
studiesyou wantto
makesurethat
luminance
distribution,
intensity and
spectralcontent
arethedesired
TheTSL2571EvaluationKitcomes
witheverything neededtoevaluatethe
TSL2571ambientlightsensor.The
evaluationkitcomprisesofamain
controller boardwithaPIC
microcontroller,anindustrystandard
USB2.0interface(withanUSBcable),
aTSL2571daughtercard,"plug-n-
play"USBHIDclassdrivers,software
documentation,andGUIsoftware
allowinguserstocontroltheALS
sensorsettingsasthePICtakesthe
TSL2571I2Cdigitaloutputsto
calculateALSilluminanceinlux
approximatingthehumaneye
response.

12. Input–sensor /
Output– actuator,trigger,stimulus,etc.
inLight/ Circadian
studiesyou wantto
makesurethat
luminance
distribution,
intensityand
spectralcontent
arethedesired
DemoKit for MAS
AS726xSpectral
sensing
https://ams.com/as726xdemokit
Multi-spectral colour sensoroptimisedfor
blue-light well-being
AMS has created a full-colour sensor AS7264N
that matches eye response with RGB, adds special
blue sensors for 440nm and 490nm, and another
fornear-infra-red.
“The sensor also accurately measures blue-light
wavelengths, which researchers have linked to
important health effects such as disruption or
management of the circadian rhythm, accelerated
eyeaging, andeyestrain.”

13. Input–sensor /
Output– actuator,trigger,stimulus,etc.
Towherearethe
animalsactually
lookingduringthe
experiment?
Poseestimation Markerlessmotioncapturesystem
(MCS)for monkeys(Macacafuscata),in
which3Dsurfaceimagesofmonkeys
werereconstructedbyintegratingdata
fromfour depthcameras(Microsoft
Kinect)
https://doi.org/10.1371/journal.pone.0166
154
Ifyouknowthegazedirection,
andtheluminancedistribution
ofthegazedirection, you
couldintegratethe“photon
dose”duringtheexperiment

14. Input–sensor /
Output– actuator,trigger,stimulus,etc.
Pose&Skeleton
estimation
Usefulformany
applicationsthen
Markerlesstrackingofuser-defined featureswithdeeplearning
AlexanderMathis, PranavMamidanna, TaigaAbe, KevinM.Cury, VenkateshN.Murthy, MackenzieW.Mathis, MatthiasBethge(
Submittedon9 Apr 2018) | https://arxiv.org/abs/1804.03142
We demonstrate the versatility of this framework by tracking various body parts in a
broad collection of experimental settings: mice odor trail-tracking, egg-laying behavior in
drosophila, and mouse hand articulation in a skilled forelimb task. For example, during
the skilled reaching behavior, individual joints can be automatically tracked (and a
confidence score isreported).
TheRolesof Supervised
MachineLearningin
SystemsNeuroscience
https://arxiv.org/abs/1805.08239
Automated leg tracking reveals distinct
conserved gait and tremor signatures
in Drosophila models of Parkinson's
DiseaseandSpinocerebellarataxia3
https://doi.org/10.1101/425405
Different mutationsproduced tremorsindistinct legpairs,
indicatingthat differentmotorcircuitsareaffected. Almost
190,000videoframes weretrackedin thisstudy,
allowing, forthefirsttime,high-throughputanalysisofgait
andtremorfeaturesin Drosophilamutants.As an efficient
assayofmutantgaitand tremorfeaturesinanimportant
modelsystem,FLLITwillenabletheanalysisofthe
neurogenetic mechanismsthat underliemovement
disorders.

15. PhysiologicalSignals
whenyou are sure thatyourenvironmentis
stableandmonitored/controlled

16. Input–sensor /
Output– actuator,trigger,stimulus,etc.
Mostlikelyyou wantto cardiac+respirationgate
themicroscopy imaging so that you are notimagingmoving tissues
RobustHeartbeatDetection fromMultimodal
DataviaCNN-based Generalizable Information
Fusion https://arxiv.org/pdf/1807.03232.pdf
SAInstruments
Sensorsforgating
SAII 1035
ECG
NIBP
Comparison ofcardiac, respiratory and dual gated images and profiles ofa
mouse heart. http://doi.org/10.1109/NSSMIC.2004.1466725
Gating(Trigger)|BIOPAC
AndPIDaswell to
keep the animalwarm
MR-compatibleFluid
Heating Systemfor
Research

17. Input–sensor /
Output– actuator,trigger,stimulus,etc.
ECGalsofor correctingintraocular pressure(IOP)
measurements
Screencaptureof20secondsofdatafromasingle,awake,unrestrainednon-humanprimate,showingIOPfluctuationsfromocular pulseamplitude,blinks,andsaccades,
whichareverysimilarinfelloweyesandcorrelatedwithorbitalmuscleactivityascapturedbytheEOGsignals.
IOP telemetry inthe nonhuman primate
J. Crawford Downs
https://doi.org/10.1016/j.exer.2015.07.015

18. ExampleoftheDAQsystemwithabitofintelligence
Newtechniquesformotion-artifact-freeinvivocardiac
microscopy
Claudio Vinegoni,SungonLee,AaronD.AguirreandRalphWeissleder
Centerfor SystemsBiology,MassachusettsGeneral HospitalandHarvardMedicalSchool,Boston,MA,USA
Front. Physiol., 12 May2015 https://doi.org/10.3389/fphys.2015.00147
Scheme of principle for motion compensation in laser scanning microscopy (LSM). (A)
DAQ, data acquisition card; ECG, electro-cardiogram; V, mechanical ventilator. (B)
Time-gated windows, coincident with the time window corresponding to the end-
diastole, are isolated in the recorded ECG. (C) In LSM images are acquired pixel by pixel
intherealspace
Scheme of principle and timing diagram for retrospectively double gated
(cardiac and respiratory) sequential segmented laser scanning microscopy. Due to
the combined effect of cardiac and respiratory motion, segments from raw images need
to be chosen in correspondence to atime-gated window, which is the intersection of two
distinct temporal windows present in the ECG and the ventilator pressure diagram.
Adapted from Lee et al. (2012a).

19. ”Simplegating” mightnotbeenough for sharp images
All-opticalmicroscopeautofocusbasedonanelectricallytunable
lensandatotallyinternallyreflectedIRlaser
M.Bathe-Peters,P.Annibale,andM.J.Lohse
OpticsExpressVol. 26, Issue 3, pp. 2359-2368 (2018)
https://doi.org/10.1364/OE.26.002359
Active motion stabilization removes relative movement between the imaging device and the imaged tissue by active motion of the objective lens
and tracking of the imaged tissue, leading to motion-free images. (A) A high-speed camera with 955 fps was utilized to track the movement
of the tissue, and a piezoactuator-driven positioner was designed for precise and fast movement of the objective lens. Adapted from
Leeetal. (2008). (B) A contact-type sensor consisting of three cantilevers beams with strain gauges was designed to measure the three
dimensional movement of the tissue instead of the previous high-speed camera. This sensor also works as a passive stabilizer, reducing the
movementwithsoftpressure.
“Active
motion
stabilization”

20. ElectricalTunable Lens(ETL) for auto-focusing
All-opticalmicroscopeautofocusbasedonanelectrically
tunablelensandatotallyinternallyreflectedIRlaser
M.Bathe-Peters,P.Annibale,andM.J.Lohse
OpticsExpressVol. 26, Issue 3, pp. 2359-2368 (2018)
https://doi.org/10.1364/OE.26.002359
We propose here a truly all-optical microscope
autofocus taking advantage of an electrically
tunable lens (ETL, Optotune EL-16-40-TC) and a
totally internally reflected infrared probe beam. We
implement a feedback-loop based on the lateral
position of a totally internally reflected infrared laser
on a quadrant photodetector, as an indicator of the
relativedefocus.
We show here how to treat the combined
contributions due to mechanical defocus and
deformation of the tunable lens. As a result, the
sample can be keptin focus withoutany mechanical
movement, at rates up to hundreds of Hertz. The
device requires only reflective optics and can be
implemented at a fraction of the cost required for
acomparablepiezo-basedactuator.
The manufacturer (Optotune) discusses only coma as a possible geometry induced
aberration in their lenses. In our hands, the dominant aberration observed was
astigmatism

21. HardwareLandscape
forgating intravital imaging

22. PlugtheSensorstothe DAQ
https://doi.org/10.1038/nprot.2015.119
RemotedevelopmentkitforFV31S-SW
FV3000IO inter-faceBox |OlympusLifeScience
6TTLoutput,6TTLinput,
4Analoginputand1Analogoutput.

23. SAIIProvidesacompletekitto imaginggating
SAII SmallAnimalInstruments,Inc.
Model1035MR-compatibleMonitor forVeterinary use
MONITORING
●
Fiber OpticECG
●
Respiration
●
Fiber OpticTemperature
●
Fiber OpticPulse
Oximetry
●
Non-InvasiveBlood
Pressure
●
InvasiveBloodPressure
●
Capnography
GATING
●
ECG
●
Respiratory
●
ECG&Respiratory
●
AuxiliaryInputs
Options include a fluid heating system which can
regulate the temperature of the animal and invasive
blood pressure measuring the cardiac waveform,
heart rate, systolic, diastolic and mean arterial
pressure..
TTL IN
ForFluoviewIOBox
*TTL
Transistor–transistor logic
HIGH/ LOW whentoimage

24. Optimally, youwould haveboth digital and analog output
TTLIN
ForFluoviewIOBox
*TTL
Transistor–transistor logic
HIGH/ LOW when toimage
ANALOGIN(s)
ForFluoviewIO
box
Usethe digitalinput duringexperiments
Butsave the rawanalogsignals aswellto
thediskalong with thedetecteddigitalHIGH /
LOWsoyou willhavesometrainingmaterialif
youwantto trainsome machine learning
for peak classificationorfor denoising,

25. Alternatives for SAII 1035: BIOPAC
BiopacDTU200
MRIGatingSystemfor TwoSignals
RespirationandECGorBP
Thesearedualchannelgatingsystemsfor small
animal.Itsendscardiactrigger pulsestotheMRI
whenarespirationsignalisinthequietphase.
Pre-processing filtersandgaincontrolsfurther
refinethequalityofthesignalandensurereliable
triggering.Includesadaptercablesfor
monitoringwithaBIOPACResearch
System.
Signal Monitoring There are outputs for the cardiac and
respiration conditioned signals (available at BNC ports:
Buffered ECG/BP and Buffered RSP) and the respective
triggers. The conditioned signals are in the ±10 volt
level range and trigger outputs are 0-5 volts. Seven
BNC to 3.5 mm monitoring cables (CBL102) and CBL122
adapters* are included.
Compatibility The unit will interface with either aBIOPAC
MP160or MP150 system. It will also work with third-party
amplifiers and data acquisition systems that operate in
the±10voltrange.
Dialsontheunitallow
conditioning of theinput
signals. Cardiacand
respiratory signals can
beamplified upto 10X.
Both inputchannelscan
belowpassfiltered
(cardiac 10-100Hz;
respiratory 1-10Hz)and
high passfiltered
(cardiac 0.1-1 Hz;
respiratory 0.05-0.5 Hz).
Conditioned signalscan
bemonitored in real
timethrough analog
inputsto theMP
system.

26. BIOPACDTU200 for Cardiac/Respiration gating
Respiration
(TSD110-MRI+
DA100CGeneral
PurposeTransducer
Amp)
ECG
ECGfrom
Electrocardiogram
Amplifier
(ECG100C/ECG100C-
MRI)
GATE-CARDRESP-Eforsmallanimal(DTU200)
Includes:
●
DualChannelCardiac RespiratoryGating System: DTU200(-E)
●
MP160/150DataAcquisition&Analysis SystemwithAcqKnowledge
software(forWindowsorMac)
●
TSD110-MRIRespiration Transducer(transducer,sensor,andtubing)
●
DA100C General-purposetransduceramplifier
●
Electrocardiography AmplifierECG100C-MRIwithleadsandelectrodes
INPUT OUTPUT

27. BIOPACDTU200 with FluoviewIO Box
Respiration
(TSD110-MRI+
DA100CGeneral
PurposeTransducer
Amp)
ECG
ECGfrom
Electrocardiogram
Amplifier
(ECG100C/ECG100C-
MRI)
INPUT OUTPUT

28. BIOPAC comes with an array ofavailable mouse sensors
The BIOPACMP160
https://www.biopac.com/application/magnetic-resonance-imaging-with-biopa
c-equipment/advanced-feature/mri-small-animal-monitoring/
System supports small animal MRI monitoring system for ECG,
Heart Rate, EMG, blood pressure, respiration, temperature, pulse
oximetry, CO2 and O2 gas analysis, electrical stimulation, and MRI
triggering. BIOPAC has a range of options that can be used in the
MRI for small animal monitoring. The modular MP160 system is
configurable to meet your exact requirements. It is also possible to
interface withexistingMRI-compatible lab equipment.

29. BIOPAC extends to wireless instrumentation (telemetry)
wirelessinvivoEEGformouse| EPOCH-R-ECG-SYS, EPOCH-M-ECG-SYS|Research| BIOPAC
https://www.biopac.com/product/epoch-wireless-ecg/

30. Electrocardiography(ECG,andPPG)
https://www.jove.com/video/1739/ambulatory-ecg-recording-in-mice
https://doi.org/10.1364/BOE.7.004313
ECGelectrodes
systems.
(a) SystemBioVet™
(©m2mImaging Corp,
Newark,USA): the
carbonfibreelectrodes
areapplieddirectlyin
contactwiththe
cleanedandshaved
chestskinandapplied
withgelelectrodeso
thataminimal
impedanceelectrical
connectionismade
withtheelectrode.
(b)Model1025small
animalmonitoringand
gatingsystem(Small
AnimalInstruments,
Inc.,StonyBrook,NY,
USA)

31. Pulse Oximetry
Images displaying the clip sensors used by
the pulse oximeter systems. (a) In the base
of the mouse or (b) in the centre ofthe footin
rat. The MouseOx®murinepulseoximeter
system from Starr Life Sciences® Corp.
(Oakmont, PA, USA) provides
measurements of O2 saturation, pulse rate,
respiration and pulse and breathe
distension. (c) ProfileofarterialO2 saturation
measurement in rat during MRI acquisitions
at 100% and 21% O2 during inhalation
anaesthesia with isoflurane.
https://doi.org/10.1186/2191-219X-2-44
Pulse Oximetry allows noninvasive monitoring of
arterial blood oxygen saturation. Fiber optic oximetry
sensors are used to transmit pulses of red and infrared
light through the animal’s peripheral vascular region.
Oxygen saturation is determined by measuring the
differentialabsorption of thered and infraredlight.
http://www.i4sa.com/web_app/main/defaultProduct.aspx?ID=34&PT=3

32. RespirationMonitoring
Minimallyinvasivehighlyprecisemonitoringof respiratory rhythm in
themouseusing an epithelialtemperatureprobe 10.1016/j.jneumeth.2016.02.007
Respiratorygating,SAII
Respiration Pad Transducer | TSD110 |
Research| BIOPAC
The TSD110 consists of a differential pressure transducer (TSD160B),
sensor (RX110), and tubing (AFT30). The TSD110 interfaces to an
MP150/MP100 via a DA100C amplifier. The Pressure Pad/Respiration
Transducer (TSD110) requires no electrical connections and works on a
numberof bodylocations(affixwith TAPE1).
https://www.biopac.com/product/pressure-pad-respiration-trans/
Extra-smallimplantsFor
usewithmiceandother
similarlysizedanimals.
DSI(divisionofHarvard
Bioscience) MouseTelemetry
https://www.datasci.com/prod
ucts/implantable-telemetry/mo
use-(miniature)

33. Respirationgating with Ventilator
Vinegonietal.(2015)
https://doi.org/10.1038/nprot.2015.119
"...Olympus microscope, and it is interfaced with a secondary
PC that records physiological and timing signals and
provides cardiac pacing capability through a custom-written
Labviewsoftwareinterface
A differential amplifier (WarnerInstrumentsDP-301) is
configured to provide a single-lead ECG ( (ADInstruments, cat.
no. MLA1213). Animal ventilation is performed with a
volume-control ventilator (ASVInspira55–7058), which
providesthesynchronizationoutput.
The secondary PC uses a data acquisition card (NI PCI-6229)
to record the animal’s ECG, as well as the analog input
synchronization signals from the microscope power supply
unit (FV10-PSU, Frame Active signal) and the ventilator
(Sync Out signal). Cardiac pacing is performed by supplying
an analog output voltage waveform to a stimulus isolator (
AMSystems,2200 stimulator) operating in voltage-to-current
conversionmode."

34. Non-Invasive Blood Pressure (NIBP) ADI#1
ADInstruments
IN125NIBPController+MLT125PulseTransducer/PressureCuff
https://www.adinstruments.com/products/nibp-systems
Requiresthe Powerlab35 DAQsystemforapower viatheI2
C
connectionforoperation, andcannotbedirectlypluggedtothe
Fluoview IOBox(oranyotherNI DAQ, etc.)

35. Non-Invasive Blood Pressure (NIBP) ADI#2
The analog inputs receive external
signals up to ±10 V. Each input has
an independently programmable
gain amplifier, filtering, and AC/DC
coupling. Set up each input with the
software, for your requirements. Input
signals can be as low as the
microvolt (µV) range without
the need for external
amplification.

36. PowerLab4/35DAQ ADI
~31,950steps ~0.0094mmHg
In theory thesmallestblood
pressurechangedetectable
AMPLIFIER+DAQ
resolution,thesensor
itselfmightbeworse,
butthis limitcannot be
exceedintheend
LSB
Leastsignificantbit
https://en.wikipedia.org/wiki/
Bit_numbering#Least_signifi
cant_bit
https://www.adinstruments.com/products/powerlab

37. IN125 with3rd
partyDAQ
ADInstruments
IN125NIBPController+MLT125PulseTransducer/PressureCuff
https://www.adinstruments.com/products/nibp-systems
GeneralPurposeTransducer Amplifier |DA100C|Research|BIOPAC

38. Non-Invasive Blood Pressure (NIBP) Biopac
A lot of the NIBP setups on the market seem to use their own
proprietary software being "dumb devices" in terms of system design
with no outputs that could be hooked directly to a DAQ (Like Fluoview
IO Box),likethe Visitech andMuromachi
Biopak seemsto havemore intelligentoptionwith the amplifier
(NIBP200A) andtail cuff(NIBP250):
https://www.biopac.com/wp-content/uploads/NIBP200A-NIBP250.pdf. https://scholar.google.co.uk/scholar?hl=en&as_sdt=0%2C5&q=NIBP200A+biopac&btnG=

39. Non-Invasive Blood Pressure (NIBP)
Unsuitable for “intelligent” contemporaryDAQ systems
The CODA tail-cuff bloodpressure systemutilizes
Volume Pressure Recording(VPR) sensortechnologyto
measurethemouseorrat tail bloodpressure. Non-
invasivebloodpressure devicesthat utilizeVPRarea
valuabletoolinresearchandwillcontinuetobebeneficial
inmanystudyprotocols.
KentDeviceDataManagementGuide
https://www.kentscientific.com/Customer-Content/www/CMS/files/Data_Manag
ement_Guide_February_2016.pdf
Your Kent Scientific Device supports a robust and
customizable set of data collection, storage and upload
features: History –stores the most recent roughly 1000
records from your runs. This data can be sent to a PC
through the USB port. Upload –uploads real-time data to
your computerthroughtheUSBport.
HarvardApparatus
BloodPressureAnalysisSystemfor
MouseandRat(SC1000)
https://www.harvardapparatus.c
om/blood-pressure-analysis-sy
stem-for-mouse-and-rat-sc100
0.html
Muromachi
MODEL MK-2000ST
NP-NIBP Monitor for Mice & Rats
https://muromachi.com/e
n/archives/english/1798/
Non-InvasiveBloodPressureSystemforRodents
HarvardApparatusPanlabNIPBsystem
https://www.harvardapparatus.com/non-invasive-blood-pressure-system-for-rodents-1.html
PressureandpulseBNCanalogsignaloutput
andRS-232serialport
“Borderlineusableasthiscomeswith
analogoutput”
BP-2000Blood Pressure
AnalysisSystemTM
http://www.visitechsystems.com/

40. RetinalImaging
additionalI/Oconsiderations

41. Youmight wouldlike to image withoutthecornealcontact
Correction-freeremotelyscannedtwo-photonin
vivomouseretinalimaging
Adi SchejterBar-Noam,NairouzFarah&ShyShoham
Light: Science & Applicationsvolume 5, pagee16007 (2016)
https://doi.org/10.1038/lsa.2016.7 → Citedby16
To scan axially without requiring the objective to come into
contact withthe cornea of theanimal, aconvex electrical tunable
lens (ETL, EL-C-10-30-VIS-LD, Optotune AG), and a concave
offset lens (−100 or −50 mm, plano-concave,Thorlabs) were
positioned in front of a 10× water immersion objective (Nikon,0.3NA,
WD = 3.5 mm). The objective lens was positioned horizontally and
coupled to the eye while the animal faced sideways (a ;→
alternatively, the objective was vertical and the eye of the animal was
facing upwards).
This analysis (c ) showed that the vast majority of available water-→
dipping objectives will be focused by the crystalline lens in
front of the retina even when the objective comes in contact with
the cornea; the only exception in our set were the low-magnification
10×objectivesfromZeiss(0.45NA,WD =1.8 mm) and from Nikon
(0.3NA, WD = 3.5 mm), and the latter provided a much wider working
range and a superior ease of use. Indeed, we were unable to
image theretina except whenusingthese objectives
Using the paraxial model, which was validated by the ray-tracing Zemax model, it
is possible to translate changes in the axial scan parameters to ‘real-world’
coordinates in the eye, which is not trivial as indicated by the 4.4 ratio between the
axial focal shifts without and inside the eye. One benefit of our approach is that it
allows for simple integration of accessory optical systems, such as
photostimulation, photo-coagulation, and optical coherence tomography (OCT),
becausetheycanbeseamlesslycombinedintothesameopticalpath.

42. ElectricalTunable Lens requiresadriver
ApplicationNote:
Opticalfocusinginmicroscopywith
Optotune’sfocustunablelensEL-10-30
https://www.optotune.com/images/products/Optotune%20application%20not
e%20for%20microscopy.pdf
TheEL-10-30canbeeasilycomputer-controlledby
usingaprecisionconstantcurrentdriverfor laser
diodes(e.g.EdmundOpticsNT56-804,Thorlabs
LD1255R,$155) anda0-250mAprogrammableanalog
output.Forsimplefocusingapplications,acalibrated
lookup-tablerelating controlcurrenttofocuspositionsis
sufficient
The Lens Driver 4 offers a simple yet precise way to control Optotune’s electrically tunable lenses, in particular the EL-
6-18 and EL-10-30 series. Communication with the driver follows an open simple serial protocol, which can be
implemented in any programming language on Windows or Linux (C#, Labview and Python source code
available). As a compact USB-powered current source, it also serves for driving LEDs or laser diodes. Comes with I2
C
sensor read-oute.g. for temperature compensation
Designed for industrial use, this LensController by Gardasoft is the ideal solution for
machine vision customers. GigE Vision, RS232 and analog inferfaces as well as
numerous SDKs allow for easy integration. The trigger input and fast response time of the
controller make it also interesting for Z-stacking in microscopy and life science
applications.
SDKs: C++, C#, VB, Labview, Cognex VisionPro, Teledyne Dalsa
Sherlock,Stemmer ImagingCVB
https://www.optotune.com/products/focus-tunable-lenses/lens-drivers
LaserDiodeDriver
DemoBoard
https://www.edmundoptics.com/p/laser-diode-
driver-demo-board-RCD-05P/39965/

43. DriverSchematics
Designed for industrial use, this LensController byGardasoft
is the ideal solution for machine vision customers. GigE
Vision, RS232 and analog inferfaces as well as numerous
SDKs allow for easy integration. The trigger input and fast
response time of the controller make it also interesting for Z-
stackinginmicroscopyand life science applications.
SDKs: C++, C#, VB, Labview, Cognex VisionPro,
TeledyneDalsaSherlock,Stemmer ImagingCVB
https://www.optotune.com/products/focus-tunable-lenses/lens-drivers
The Lens Driver 4 offers a simple yet precise way to control Optotune’s electrically tunable
lenses, in particular the EL-6-18 and EL-10-30 series. Communication with the driver follows an
open simple serial protocol, which can be implemented in any programming language on
Windows or Linux (C#, Labview and Python source code available). As a compact USB-
powered current source, it also serves for driving LEDs or laser diodes. Comes with I2
C sensor
read-out e.g. for temperature compensation
CONSTANT
CURRENT
DRIVE
https://www.optotune.com/images/products/Optotune%20Lens%20Driver%204%20manual.pdf
https://www.optotune.com/Gardasoft_TR_CL180_Datasheet_v001.pdf
One channel, including constant current lens drive and lens EEPROM data communications.
Automatically reads data from EEPROM inside lens which calibrates the controller response.
The performance of the controller istherefore automaticallytailored toeach individual lens.

44. ExampleSetupsandStudies

45. HighFrame Rates forgood images
Invivomultiphotonmicroscopyof cardiomyocyte
calciumdynamicsinthe beatingmouseheart
Smalletal.(2018)https://doi.org/10.1101/251561
(b) Electrocardiogram (ECG) and ventilator pressure are recorded simultaneously during image acquisition
allowing image reconstruction. Red vertical lines indicatethe start of each frame; red arrow indicates the
peak of R-wave used as the start of the cardiac cycle for the frame displayed below; blue arrow indicates
the end of respiratory exhalation that was used as the marker of respiratory cycle. (c) Single raw image
frames with colored boxes indicating the image segments, with corresponding timing of the acquisition
indicated on the ECG and ventilator pressure traces. (d) A plane reconstructed using 512 x 33 pixel
segments, 5% of the cardiaccycle, restricted to 70-100% of therespiratory cycle, and averaged across 4 µm
in z.
We demonstrated intravital multiphoton microscopy in the beating
heart in an intact mouse and optically measured action potentials with
GCaMP6f, a genetically-encoded calcium indicator. Images were
acquired at 30 fps with spontaneous heart beat and continuously
runningventilatedbreathing.
Higher frame rate imaging shows reduced in-frame motion due to heart contraction. Raw
image frames showing same cardiac vessel with (a) standard galvonometric scanning
and (b) resonant scanning. Green dotted lines indicate the timing of the peak of the R wave from the
electrocardiogram which align with image artifacts.
Resonant scanning (Cambridge Technology) data acquisition was performed using a National
Instruments digitizer (NI-5734), FPGA (PXIe-7975), and multifunction I/O module (PXIe-6366) for
device control, mounted in a PXI chassis (PXIe-1073) controlled by ScanImage 2016b. A Ti:Sapphire
laser (Chameleon, Coherent) with the wavelength centered at 950 nm, was used to simultaneously
exciteGCaMP6fand Texas-Red fluorescence.
ECG and respiratory voltage signals were collected with the two unused detection channels
allowing simultaneous recording during imaging. A series of 50–100 frames (1.7 to 3.3 s) per plane in z
were collected at ascanspeed of 30frame/sec. Assigningcardiac and respiratory phase toimage.
We found that with a heart rate of about 5 Hz and breathing at 2 Hz, ~1.5 seconds or about 50
frames was sufficient to generate images in most of the cardiac/respiratory cycle phase space.
Matlab was used for reconstruction and cardiac/respiratory phase-dependent analysis. Scripts
areavailable in Supplement Materials.

46. Gating incardiovascularmicroscopy
Multi-photonmicroscopyincardiovascularresearch
Wuetal.(2017)http://dx.doi.org/10.1016/j.ymeth.2017.04.013
Motional artifacts and loss of focus in un-triggered in vivo TPLSM imaging. The blood
pressure variation during systole and diastole causes vessel contraction and relaxation, resulting in
intra-frame and inter-frame (out-of-focus) artifacts in the images. Three subsequent optical
sections of left carotid artery obtained in vivo without application of external triggering.
Frame rate was 2.3 Hz (1200 lps; line scan rate 1X, image size 400 * 400 pixels). Cell nuclei are visible.
Bars indicate 50 µm. Images are disturbed by intra-frame motional artifacts, causing the arterial wall to
appear as a curved-like structure. Inter-frame artifact (out-of-focus images) due to respiratory
movement result in a different imaging depth of the blood vessel, depending on the phase in the
cardiaccycle.During un-triggeredinvivoimaging, in focusimagesarerarely acquired.
Examples of intravital atherosclerosis (A-E) imaging. A) Imaging of major arteries after endothelial injury (dashed
lines show theoutlineof theelastin layers,) showing cell debrison theluminal sideof theblood vessel (whitearrows) and
subendothelial expression of the inflammatory marker VCAM-1-AF568 (red) in comparison to B) a healthy blood vessel
with an intactendothelial layer (green), labeled using CD31- AF488. C) Both collagen and elastin can beimaged without
labeling, using autofluorescence (coded green) or SHG (coded red), repetitively. These structures can be visualized
better after the addition of dyes, e.g., D) sulfo-rhodamine B (red) for elastin (white arrow) or E) CNA35-FITC (green) for
collagen in plaque-containing carotid artery. Enhanced accumulation of collagen can be observed in the plaque
shoulderregion.

47. ProspectiveGating incardiovascularmicroscopy
Sequentialaveragesegmentedmicroscopyforhighsignal-to-
noiseratiomotion-artifact-freeinvivoheartimaging
ClaudioVinegoni,SungonLee,PaoloFumeneFeruglio,PasquinaMarzola,
MatthiasNahrendorf,andRalphWeissleder
BiomedicalOpticsExpressVol.4,Issue10,pp.2095-2106(2013)
https://doi.org/10.1364/BOE.4.002095
Schemeofprincipleforsequentialretrospective
electrocardiogram(ECG)-gatedsegmentedmicroscopy.For
figuresimplicity,weassumeheretheabsenceofany
respiratorymotion.
(c) Prospectivetriggered acquisitionscheme: datafor
imagesareacquiredonlyduringthetimeofaspecific
triggeredwindow,whichisdeterminedbyECG.Allacquired
dataarethereforeusedforimagereconstruction.
(d) Retrospective gatedacquisitionscheme:datafor
imagesarecontinuouslyacquiredtogether withtheECG
recording.Followingthisnon-selectiveacquisition,onlythe
datathatwereacquiredduringthetimeofaspecificgated
window,whichisdeterminedbyECG,arechosenforimage
reconstruction.RRindicatesthedistancebetweentwoR
phases.IMindicatesagenericimage.

48. Gating incardiovascularMRIimaging#1
Real-TimeGatingSystemforMouseCardiovascularMRImaging
MaherSabbah,HasanAlsaid,LatifaFakri-Bouchet,CedricPasquier,Andre
Briguet,EmmanuelleCanet-Soulas,andOdetteFokapu
MagneticResonanceinMedicine57:29–39(2007)
https://doi.org/10.1002/mrm.21096 |Citedby13
High-resolution MR images of mouse hearts and aortic arches were
acquired using a chain consisting of ECG signal detection, digital signal
processing, and gating signal generation modeled using Simulink (The
MathWorks,Inc.,Natick,MA,USA).
The signal-processing algorithmsusedwererespectivelylow-passfiltering,
nonlinear passband, and wavelet decomposition. Both updated and
nonupdated gating signal generation methods were tested. Noise
reduction was assessed by comparison of the ECG signal-to-noise ratio
(SNR) before and after each processing step. Gating performance was
assessed by measuring QRS detection accuracy before and after online
trigger-leveladjustments.
Low-pass filtering with trigger-level adjustment gave the best
performance for mouse cardiovascular imaging using gradient-echo
(GE), spin-echo (SE), and fast SE (FSE) sequences with minimum induced
delay and maximum gating efficiency (99% sensitivity and R-peak
detection).
This simple digital gating interface will allow various gating strategies to be
optimizedfor cardiovascularMRexplorationsinmice.
Further studies
willseekto
validate
cardiorespiratory
gating withreal-
timeextractionof
therespiratory
signalfromthe
respiration-
modulatedECG
signal.

49. “Intelligent”Gating incardiovascularMRIimaging
Prospectivegatingcontrolforhighlyefficientcardio-
respiratorysynchronisedshortandconstantTRMRIinthe
mouse
PaulKincheshetal.
MagneticResonanceImagingVolume53,November 2018,Pages20-27
https://doi.org/10.1016/j.mri.2018.06.017
Where steady state imaging techniques are required in small animals,
synchronisation is most commonly performed using retrospective
gatingtechniquesbuttheseinvokeaninherenttimepenalty.
Prospective gating incorporating the automatic reacquisition of
data corrupted by motion at the entry to each breath was implemented in
short TR 3D spoiled gradient echo imaging. Motion sensitivity was
examined over the whole mouse body for scans performed without
gating, with respiratory gating, and with cardio-respiratory gating. The
gating methods were performed with and without automatic reacquisition
ofmotioncorrupteddataimmediatelyaftercompletionofthesamebreath.
Diagrammatic representation of respiration gated (R-gated) and cardio-respiratory gated (CR-
gated) MRIschemes. Threshold levels are set on the amplified and filtered ECG and respiration (Resp)
analogue voltages to generate the C-logic and R-logic control signals respectively. The R-logic control
signal is evaluated for R-gated scanning. A user-variable post breath delay ( )τ) is used to ensure that
motion artefact is minimised from the trailing portion of the breath. Only the C-logic signalsthat occur during
the R-logic high level gate are selected to generate the CR-logic control signal which is evaluated for CR-
gated scanning. In the diagram a single respiration corrupted data acquisition block
(marked CD) is automatically reacquired as soon as each breath completes
(markedRD)toreduceartefactfrommotionduringtheonsetofeachbreath.

50. Gating forhumanMRI
Physiorack:AnintegratedMRIsafe/conditional,Gasdelivery,
respiratorygating,andsubjectmonitoringsolutionfor
structuralandfunctionalassessmentsof pulmonaryfunction
J.Magn.Reson.Imaging2014;39:735–741 TechnicalNote
AhmedF.HalaweishPhD H.CecilCharlesPhD
https://doi.org/10.1002/jmri.24219
Actual setup of Physiorack components both inside the scanner room (a) and in the
control room (b), as would be implemented during any given imaging session. (Not in
picture: Oro-nasalfacemask,filtersandPulseoximetrysystem.)
The signals recorded from the pneumotach transducers are amplified by means of
transducer amplifier modules (Biopac, Model DA 100C). All sampled signals
(respiratory, gaseous concentrations, pulse-oximetry, etc.) are recorded and digitized
using a pair of digitizing acquisition modules (DAQ, Windaq, Model DI-158, DataQ
Instruments,Akron,OH).
To evaluate the use of a modular MRI conditional respiratory
monitoring and gating solution, designed to facilitate proper
monitoring of subjects' vital signals and their respiratory efforts, during free‐
breathing and breathheld 19F, oxygen enhanced, and Fourier‐ ‐
decompositionMRI basedacquisitions.‐
We demonstrate an inexpensive,off the shelfsolutionfor monitoring these‐ ‐
signals, facilitating assessments of lung function. Monitoring of
respiratory efforts and exhaled gas concentrations assists in
understanding the heterogeneity of lung function visualized by gas
imaging.

51. Active motion measurement
Motioncharacterizationschemetominimizemotionartifactsin
intravitalmicroscopy
Leeetal.(2017)https://doi.org/10.1117/1.JBO.22.3.036005
METHODS: During intravital imaging sessions,
mice were anesthetized with 2% isoflurane and
2 l/min oxygen, and the body temperature of the
mice was kept constant at 37°C during all
procedures (surgery and imaging). For mice
ventilation, an animal ventilator (Harvard
Apparatus INSPIRAASV55-7058) was used.
The ECG signal, recorded using three needle
electrodes subcutaneously placed in the two
front legs and the right hind leg, was filtered and
amplified using a differential preamplifier (
ADInstrumentsDP-301, output ±10 V). Both
ECG and ventilator traces were recorded using
a data acquisition card (DAQ) (National
Instruments, NI PCI-6229, 1600 SGD, needs the
BNC block NIBNC-2110, 600 SGD) and
Labviewsoftware.
For sensing, a submicron-precision laser
displacement sensor unit (KeyenceLG-030,
~2000SGD) was mounted onto an objective
holder sliding nosepiece, allowing to easily
switch between the imaging objective and the
sensor,withoutrepositioningtheimagedanimal.
In vivo motion characterization. (a) A typical example
of tissue movement as measured by the custom-made
motion characterization system. Two dominant
repetitive motions are observed. The one with big
amplitude is due to respiration and the small one due
to cardiac activity. The ECG measurement in green
color confirms that the small movement is caused by the
heartbeat, and it issynchronized withmotion.

52. Real-time operatingsystems withDAQ
Real-TimeLinuxDynamicClamp:AFastandFlexibleWayto
ConstructVirtualIonChannelsinLivingCells
AlanD.DorvalDavidJ.ChristiniJohnA.White
AnnalsofBiomedicalEngineeringOctober 2001,Volume29,Issue10,
https://doi.org/10.1114/1.1408929
“The dynamic clamp require a high frequency current
clamp amplifier. The amplifier must connect to a
personal computer (PC) controlled, data acquisition
board (DAQ). Our amplifier was connected to a National
Instruments, PCI-MIO-16XE-50 data acquisition
board. This DAQ boasts 16 channel, 16 bit analog-to-
digital input (A/D) and 2 channel, 12 bit digital-to-analog
output(D/A),bothrunningatamaximumof20kHz.”
“The PC runs a free, open source extension to the Linux
operating system, known as Real Time Linux (RTL).
RTL is a ‘‘hard’’ real time operating system, which
means that commands will always be executed in a
known amount of time. RTL provides high temporal
precision on a PC, while maintaining the full functionality
of the now widely supported parent operating system,
Linux.”

53. DAQA/D Resolution importance
Performancecomparisonbetween8-and14-bit-depthimaginginpolarization-
sensitiveswept-sourceopticalcoherencetomography
ZenghaiLu,DeepaK.Kasaragod,andStephenJ.Matcher (2011)
https://doi.org/10.1364/BOE.2.000794
AQplusreceivernoise
measurementsatdifferentset
fullanaloginputvoltageranges
(FIVR) for 14-bit(a)and8-bit
DAQ(b),respectively.(c):
standarddeviationofthe
measuredDAQplusreceiver
noisealongwiththecalculated
noisestandarddeviationof
quantizationnoiseoftheDAQ
https://spectrum-instrumentation.com/en/m2i4022
We compare true 8- and 14-bit-depth
imaging of SS-OCT and polarization-
sensitive SS-OCT (PS-SS-OCT) by
using two hardware-synchronized high-
speeddataacquisition (DAQ)boards.
The two signals are sampled at 20MS/s
simultaneously with 14-bit (M2i.4022,
Spectrum GmbH, Germany) and 8-bit (
M2i.2031, Spectrum GmbH, Germany)
resolution.

54. The “IntelligentDAQs”

55. Startwithopen-sourceplatforms suchasRTXI
Hardreal-timeclosed-loopelectrophysiologywiththeReal-
TimeeXperimentInterface(RTXI)
YogiA.Patel,AnselGeorge,AlanD.Dorval,JohnA.White,DavidJ.Christini,
RobertJ.ButeraPLOSComputationalBiology13(7):e1005656
https://doi.org/10.1371/journal.pcbi.1005430
https://doi.org/10.1371/journal.pcbi.1005656
http://rtxi.org/
https://github.com/rtxi
On-going RTXI development efforts are also
focused on providing API calls for distributing
computational loads across dedicated
processor cores and GPUs, with the goal of
requiring little to no technical know-how on the
user’send.
RTXI uses the open source Xenomai framework to implement
communication with a variety of commercially available multifunction
DAQ cards with both analog and digital input and output channels. This
makes RTXI essentially hardware-agnostic and able to
communicate with multiple actuators and sensors that may span different
modalities.
ListofDAQssupportedbytheanalogydriver
Driverslist ni_pcimio
This drivers suppors a long list of NationalInstrumentsPCI /PXI
cards:
PCI-MIO-16XE-50, PCI-MIO-16XE-10, PCI-MIO-16E-1, PCI-MIO-16E-4,
PCI-6014
PCI-6023E, PCI-6024E, PCI-6025E, PXI-6025E
PCI-6030E, PXI-6030E, PCI-6031E, PCI-6032E, PCI-6033E, PCI-6034E,
PCI-6035E, PCI-6036E
PCI-6040E, PXI-6040E
PCI-6052E, PXI-6052E
PCI-6070E, PXI-6070E, PCI-6071E, PXI-6071E
PCI-6110, PCI-6111
PCI-6220, PCI-6221
PCI-6143,PXI-6143
PCI-6224, PCI-6225, PCI-6229
PCI-6250, PCI-6251, PCIe-6251,PCI-6254, PCI-6259, PCIe-6259
PCI-6280, PCI-6281, PXI-6281, PCI-6284, PCI-6289,
PCI-6711, PXI-6711,PCI-6713,PXI-6713,
PCI-6731,PCI-6733, PXI-6733,

56. GPUswithDAQs convergingwithreal-timedeeplearning
DataAcquisitionwithGPUs:TheDAQfortheMuong-2
ExperimentatFermilab
W.Gohn(Submittedon15Nov2016)
https://arxiv.org/abs/1611.04959
The muon g-2 experiment at Fermilab is heavily
relying on GPUs to process its data. The data
acquisition system for this experiment must have
the ability to create deadtime-free records from
700 µs muon spills at a raw data rate 18 GB
per second. Data will be collected using 1296
channels of µTCA-based 800 MSPS, 12 bit
waveform digitizers and processed in a layered
array of networked commodity processors with
24 GPUs working in parallel (26 Nvidia Tesla
K40 GPUs housed by pairs in 13 front-end
computers) to perform a fast recording of the
muondecaysduring the spill.
In addition to numerous models of GPUs, there are also coprocessor systems such
as the Intel Xeon Phi, which utilize fewer but faster cores than the GPUs, as well as
FPGAsor ASDQs,which require significantlymore programming overhead than do
theGPUssystems
18 GB per second
→ 144 Gbit/s
Comparetohigh-speedcameras
with PCIExpressGen.3x8 with8
GBperseconddatarates

57. GPUswithDAQs withorasanalternativetoFPGAs
GPUforDAQ triggering:feasibilitystudy
PhilipRodrigues,UniversityofOxford
October19,2017
https://indico.fnal.gov/event/15558/contribution/2/material/slides/0.pdf
Bandwidthbottlenecks
PCIe4.0approvedrecently,offers30GB/s.
Nvidia’sproprietary“NVLink”availableinhigh-endservers, 150GB/s
IntroductiontoGPUComputingwithLabVIEW
NationalInstruments,Aug 2,2013
http://www.ni.com/white-paper/14077/en/
Using the LabVIEW GPU Analysis Toolkit, developers have the ability to offload
significant calculations to a GPU for processing, freeing up the CPU to work on
other tasks. This affords a LabVIEW user a very powerful processing resource that
was not previously available. Acquired data can now be rapidly processed using not
only FPGAs and CPUs, but also GPUs, and viewed from a single LabVIEW
application.
https://slideplayer.com/slide/7422328/
Samvan derJeught

58. FPGAs aspre-processorsfor GPUswithDAQs
Low-latencydataacquisitiontoGPUsusingFPGA-based 3rdpartydevices
DenisPerret
LESIA/ObservatoiredeParis
~ StratixV(15k USD)PCIedevelopmentboardfromPLDA(+ QuickPCIe,
QuickUDP IPcores)42Gb/sdemonstratedfromboardto GPU;8.8 Gb/s
per10GbE link inloopbackmode
ExtremelyLargeTelescope(ELT)withAdaptive
Optics(AO) correction

59. GPUacceleratedDAQswithOCTimagingaswell#1
DevelopingtheWorld’sFirstReal-Time3DOCT Medical
ImagingSystem WithLabVIEWandNI FlexRIO
Dr.KohjiOhbayashi 大林 康二 ,KitasatoUniversity,GraduateSchoolofMedicalScience
http://sine.ni.com/cs/app/doc/p/id/cs-13387
“Using optical coherence tomography (OCT) and a 320-
channel data acquisition system combining NI FlexRIO field-
programmable gate array (FPGA) hardware and GPU
(NVIDIA Quadro FX 3800) processing to create the world’s
firstreal-time 3D OCT imaging system”
For high-speed acquisition, we use the NI 5751 adapter module,
which has a 50 MS/s sample rate on 16 simultaneous channels with 14-
bit resolution. The adapter module interfaces to the NI PXIe-7962R
FPGA module, which we use to perform the first stage of processing –
subtraction of the sample-cut noise and multiplication of a window
function. In total, we have 20 modules across two PXI Express chassis,
so we use two NI PXIe-6674T timing and synchronization
modules to distribute clocks for the system and assure precise phase
synchronizationacrossallthechannelsinthesystem.

60. GPUacceleratedDAQswithOCTimagingaswell#2
AlazarTech
ATS9373DAQ+ ATS GMA‑GMA +GPUAMDRadeonProGraphicalProcessingUnit
April2018
https://www.alazartech.com/landing/oct-news-2018-04

61. GPUacceleratedDAQswithOCTimagingaswell#2
High-speedFPGA-GPUprocessingfor3D-OCT imaging
Kyung-ChanJin; Kye-SungLee; Geun-HeeKim(March 2018)
https://doi.org/10.1109/CompComm.2017.8322904
In thispaper, we propose the designofa real-time image acquisition andpre-
processing FPGA(NI PCIe-1473R)viaLabVIEW(NationalInstruments(NI))with
GPU-basedaccelerationthatiscapableofsustainingtherateofdataacquisition.
Results showed that, by applying GPU acceleration to the tomographic
inspectionofbiologicalsamples,SD-OCTimaginginexcessof40frames/s(FPS)
for the NVIDIA M6000 (7 Tflops at fp32) GPU-accelerated SD-OCT with frame
size 4096 (axial) × 512 (lateral) becomes feasible, and more than 512 × 512 × 500
volumes can be reconstructed with a speed increase of at least 7x that of a
non-GPU.
Linux-based
systemwithFPGA-
GPUmodule
we utilized the Spimagine Python packageto interactively visualize
and process the 3D tomographic image (via OpenCL)

62. Massivedatarates possiblealsowithmicroscopy
LCLS2DataReduction
PipelinePreparationsfor
SerialFemtosecond
Crystallography
Chuck Yoon
HDRMX,Mar 16,2017
https://slideplayer.com/slide/12645601/
Integrate somedenoisingwithdeep-learningbasedcompressionforreducingdatawrittenon disk?

63. Goforhighenergyphysicists forinspiration?
DAQ/FEE/Trigger forCOMPASSbeyond 2020workshop
https://indico.cern.ch/event/673073/timetable/?print=1&view=standard | https://indico.gsi.de/event/7173/
Thisworkshopisfocusedondevelopmentneededfor COMPASSbeyond2020.Wewill discussrequiredperformanceand
architectureofFront-EndElectronics(FEE) andDAQcomponents,unifyserialinterfacesandprotocols,discuss triggerprocessor
hardware,anddistributionofworkload.One ofthetopicswill bealso developmentofSidetectorsystemsforpolarizedtarget.
Moreinfocanbefoundat workshopwebpage
Overviewabouttriggerhardware
BenjaminMoritzVeit (JohannesGutenberg Universitaet Mainz(DE))
bveit_DAQFEET2017_trigger_hardware.pdf
FPGAbasedtriggerdevelopment
DmytroLevit (TechnischeUniversitaetMuenchen(DE))
trigger.pdf

64. IntelligentParadigms

65. You mightwanttoadjustyourstimulusbasedonresponse
Neurofeedback paradigms with brain stimulation (tACS, rTMS), steady-state visual evoked responses (SSVEPs),
individualized alphafrequency (IAF) driving, etc.
D. Reatoet al. . Effectsof weak
transcranial alternatingcurrent
stimulation on brain activity—areview of
knownmechanismsfrom animal studies.
FrontiersinHumanNeuroscience,7,Oct.2013.
http://dx.doi.org/10.3389/fnhum.2013.00687
Gets even tricker ifyou need to read neuron firing
from an image(calcium dye) in real-time as you need
somedeep learning imageanalysis for this.
Frequency dependenceof optogenetic
slicemodeloftACSfrom Kukietal.2013
LeChasseuretal.(2011)
Electrophysiology withoptical
electrocorticography

66. “SpatiotemporalElectrophysiology” ImageAnalysisneeded
Lindetal.(2013)
Lindetal.(2013)
Controlsystemparameter valuesplottedagainsttheageof
eachvolunteer. NeurovascularCouplingRemains
UnaffectedDuringNormalAging(2003)
Maybe you need to just set
individualized parameters just before
each individual without need for real-time
tracking
Alpha neurofeedbacktrainingimproves
SSVEP-basedBCIperformance
https://doi.org/10.1088/1741-2560/13/3/036019

67. SpatiotemporalNeurovascular Coupling
Functionalopticalcoherencetomographyofneurovascular
couplinginteractionsintheretina
Sonetal.(2018)https://doi.org/10.1002/jbio.201800089
Here, we report a multimodal functional optical coherence
tomography (OCT) imaging methodology to enable
concurrent intrinsic optical signal (IOS) imaging of stimulus‐
evoked neural activity and hemodynamic responses at
capillaryresolution.
OCT angiography guided IOS analysis was usedto separate
neural IOS‐IOS and hemodynamic IOS changes‐IOS in the
same retinal image sequence. Frequency flicker stimuli
evoked neural IOS changes in the outer retina‐IOS ; that is,
photoreceptor layer, first and then in the inner retina, including
outer plexus layer (OPL), inner plexiform layer (IPL), and
ganglion cell layer (GCL), which were followed by
hemodynamic IOS changes primarily in the inner‐IOS
retina;thatis, OPL,IPL,andGCL.
Different time courses and signal magnitudes of
hemodynamic IOS responses were observed in blood‐
vesselswith variousdiameters
Alteredneurovascularcouplingasmeasuredbyoptical
imaging: abiomarkerfor Alzheimer'sdisease
https://doi.org/10.1038/s41598-017-13349-5
Spatiotemporal
patterns analyzed
now afterthe
experiment, but what
if you would want to
do this in real-time?

68. Goforserious BigData techniqueslikeApacheSpark

69. “BigData” enteringvolumetricfunctionalimaging
StudyingAxon-AstrocyteFunctionalInteractionsby3DTwo-
PhotonCa2+Imaging:APracticalGuidetoExperimentsand
“BigData”Analysis
IaroslavSavtchouk,GiovanniCarrieroandAndreaVolterra
Front.Cell.Neurosci.,13April2018
https://doi.org/10.3389/fncel.2018.00098
Recent advances in fast volumetric imaging have enabled
rapid generation of large amounts of multi-dimensional
functional data. While many computer frameworks exist for
data storage and analysis of the multi-gigabyte Ca2+
imaging experiments in neurons, they are less useful for
analyzing Ca2+ dynamics in astrocytes, where
transients do not follow a predictable spatio-temporal
distributionpattern.
In this manuscript, we provide a detailed protocol and
commentary for recording and analyzing three-dimensional
(3D) Ca2+ transients through time in GCaMP6f-expressing
astrocytes of adult brain slices in response to axonal
stimulation, using our recently developed tools to perform
interactive exploration, filtering, and time-correlation analysis
ofthetransients.
https://wwwfbm.unil.ch/dnf/group/glia-an-active-synaptic-partner/member/v
olterra-andrea-volterra
form ofsoftwarepluginsfor ImageJ (NIH)
Synchronization of imaging and electrical stimulation via simultaneous capture of timing
information from the two systems. By concurrently recording the image frame counts and the
electrical inputs, one can later link analytically the imaging and electrophysiology data. (A) Schematic
diagram of connections. An electrophysiology computer is also simultaneously recording two signals
for synchronization: a Y-galvanometer position feedback pin, and a split-off of a stimulator TTL
trigger. (B) Low-zoom overview of the captured synchronization signals (light-blue highlight is
magnified in next panel, not to scale). Vertical deflections in the Y-galvo trace correspond to individual
Y-frame scans, whereas spikes on the Stim TTL trace indicate the relative timings of axonal
stimulation (C). A high-zoom version of the highlighted stretch in (B), showing the relationship
between the imaging frame position and the stimulation signal timing. Each Z-stack consumes an
entire Y-scan frame per focal plane (here 21) plus any additional overhead, depending on whether bi-
directionalz-scanningisimplemented,etc.

70. Closed-LoopElectrophysiology Example #1
Hardreal-timeclosed-loopelectrophysiologywiththeReal-
TimeeXperimentInterface(RTXI)
YogiA.Patel,AnselGeorge,AlanD.Dorval,JohnA.White,DavidJ.Christini,
RobertJ.ButeraPLOSComputationalBiology13(7):e1005656
https://doi.org/10.1371/journal.pcbi.1005430
https://doi.org/10.1371/journal.pcbi.1005656
Real-time control applications for biological research are
available; however, these systems are costly and often restrict
the flexibility and customization of experimental protocols. The
Real-Time eXperiment Interface (RTXI) is based on Xenomai, a
Real-Time Linux framework, and is an open source software
platform for achieving hard real-time data acquisition and
closed-loop control in biological experiments while retaining the
flexibilityneededfor experimental settings.
RTXI has enabled users to implement complex custom closed-loop
protocols in single cell, cell network, animal, and human
electrophysiology studies. RTXI is also used as a free and open
source, customizable electrophysiology platform in open-loop
studies requiring online data acquisition, processing, and
visualization.
RTXI interfaces with experiments through a variety of hardware interfaces,
including PCI/PCIebasedDAQsfromNational Instrumentsand Sensoray, Ethernet
based devices such as cameras and commercial amplifiers, as well as USB-based
acquisitiondevice
On-going development efforts within RTXI are focused on incorporating new measurement
modalities (e.g., imaging) and acquiring from high channel count interfaces—all with hard RT
closed-loop performance. For example, a image acquisition and processing module (GenICam)
andaEthernet-based dataacquisition module(EthernetAcq)arenowavailableforusewithin RTXI.In
many cases, hard RT closed-loop control with image or high channel count data processing can
require more computation time than is available per cycle. On-going RTXI development efforts
are also focused on providing API calls for distributing computational loads across
dedicated processor cores and GPUs, with the goal of requiring little to no technical know-how
on theuser’send.
Experiment data and metadata saved by the
Data Recorder are stored in the Hierarchical
Data Format (HDF5). HDF5 file read and write
operations are supported by many common
analysis frameworks and languages (e.g.,
MATLAB, Python, Julia, R, etc).

71. Closed-LoopElectrophysiology Example #2
ClosedLoopExperimentManager(CLEM)—AnOpenand
InexpensiveSolutionforMultichannelElectrophysiological
RecordingsandClosedLoopExperiments
HananelHazanandNoamE.Ziv
Front.Neurosci.,18October2017
https://doi.org/10.3389/fnins.2017.00579
https://github.com/Hananel-Hazan/CLEM-Plugin-Template
Here,wesurveycommercialandopen
sourcesystemsthataddresstheseneeds
tovaryingdegrees.Wethenpresentour
ownsolution,whichwerefer toasClosed
LoopExperimentsManager(CLEM).
CLEMisanopensource,softreal-time,
MicrosoftWindowsdesktopapplication
thatisbasedonasinglegenericpersonal
computer (PC)andaninexpensive,
general-purposedataacquisition
board (PD2-MF-64-3M/12,United
ElectronicsIndustries).CLEMprovidesa
fullyfunctional,user-friendlygraphical
interface,possessesfacilitiesfor recording,
presentingandlogging
electrophysiologicaldatafromupto64
analogchannels,andfacilitiesfor
controlling externaldevices,suchas
stimulators,throughdigitalandanalog
interfaces.
Microsoft Windows is not a real-time OS; most contemporary data acquisition boards, however, provide
asynchronous acquisition modes in which the board transfers incoming data into PC memory autonomously.
Typically, interrupts are generated each time the transfer of predetermined numbers of samples is completed.
These events can be used to clock sample-analyze-output loops. We found that this mode is sufficiently
reliabletosupportclosedloopperformancewithlatenciesunder2mswithpracticallynojitter

72. Closed-LoopElectrophysiology Example #3
Multimed:AnIntegrated,Multi-ApplicationPlatformforthe
Real-TimeRecordingandSub-MillisecondProcessingof
Biosignals
AntoinePirog,YannickBornat,RomainPerrier,MatthieuRaoux,Manon
Jaffredo,AdamQuotb,JochenLang,NoëlleLewis,andSylvieRenaud
Sensors(Basel).2018Jul;18(7):2099.
https://dx.doi.org/10.3390%2Fs18072099
We designed Multimed, which is a versatile hardware platform
for the real-time recording and processing of biosignals. Digital
processing in Multimed is an arrangement of generic processing
units from a custom library. These can freely be rearranged to
matchthe needsof the application.
Embedded onto a Field Programmable Gate Array (FPGA), these
modules utilize full-hardware signal processing to lower processing
latency. It achieves constant latency, and sub-millisecond
processing and decision-making on 64 channels. The FPGA core
processing unit makes Multimed suitable as either a
reconfigurable electrophysiology system or a prototyping
platform for VLSI implantable medical devices. It is specifically
designed for open- and closed-loop experiments and
provides consistent feedback rules, well within biological
microsecondstimeframes.
Multimedsetupshave been installedin multiplesitesandare being usedbypartner laboratories
in collaborative projects, always aiming for closed-loop experiments (BRAINBOW (EU
project 284772, ICT- FET FP7/2007–2013, FET Young Explorers) [33], CENAVEX (ANR grant
2013-NEUC-0001-01 and NIH grant 5 R01 NS086088-02) [34,35,40], ISLET CHIP (ANR grant
2013-PRTS-0017)[25,41],HYRENE(ANRgrant2010-BLANC-0316-01)[21])

73. BeyondImmediateResearchGoals
Get datafor image restoration,
reconstruction, and “technical
optimization” purposesaswell.
i.e.datatotraindeeplearning
networks

74. Deeplearningcanimprovethemicroscope imagequality
DeeplearningmicroscopyYairRivenson,ZoltánGöröcs,Harun Günaydin, Yibo Zhang,Hongda Wang,and Aydogan Ozcan.
Optica Vol.4, Issue11, pp. 1437-1443 (2017) https://doi.org/10.1364/OPTICA.4.001437
Thefirststepin thisdeep-learning-
basedmicroscopyframework
involveslearningthestatistical
transformationbetweenlow-
resolutionandhigh-resolution
microscopicimages,whichis used
totrainaCNN
Wehavechosenbright-field
microscopy withspatiallyand
temporallyincoherentbroadband
illuminationasanexample, the
samedeeplearningframework
mightbeapplicabletoother
microscopy modalities,including,
e.g.,holography,dark-field,
fluorescence,multi-photon,optical
coherence tomography,among
others.
After appropriate training, this framework and its derivatives might
be applicable to other forms of optical microscopy and imaging
techniques and can be used to transfer images that are
acquired under low-resolution systems into high-
resolution and wide-field images, significantly extending the
space bandwidth product of the output images. Furthermore,
using the same deep learning approach we have also
demonstrated the extension of the spatial frequency response of
the imaging system along with an extended DOF. In addition to
optical microscopy, this entire framework can also be applied to
other computational imaging approaches, also spanning
different parts of the electromagnetic spectrum, and can be used
to design computational imagers with improved resolution, FOV,
and DOF.

75. Youcouldhavea“goldstandard”systemwithhigh-endgatingand
motionmeasurement/compensationsystem
Andthentrain
deeplysupervised
deeplearning
networktoboth
detectandcorrect
motionartifacts?
Deep learning-based detectionofmotionartifactsin
probe-based confocal laserendomicroscopyimages
MarcAubreville,MaikeStoeve,NicolaiOetter,MiguelGoncalves,ChristianKnipfer,Helmut
Neumann,ChristopherBohr,FlorianStelzle,AndreasMaier
InternationalJournalofComputerAssistedRadiologyandSurgery(2018)
https://doi.org/10.1007/s11548-018-1836-1
Each of the images was manually assessed for motion artifacts by
two experts with background in biomedical engineering, while the second
expert was able to see the annotations of the first expert (non-blinded). The
annotation results have been validated by two medical experts with profound
experience in CLE diagnosis. All annotations (bounding boxes of
artifacts) werestored in arelational databaseandusedforbothtrainingand
evaluation.
Efficient DataLabellingfor Ocular Imaging
https://www.slideshare.net/PetteriTeikariPh
D/efficient-data-labelling-for-ocular-imagin
g-110540104

76. Ifyoursystemisfastenough,youcouldmeasureboth“motion-free”and
“motion”periodsofthetissue,andlearnthe blur/motionkernel?
Your groundtruth
imagewith “no blur”
Blurredimages,
unknown PSFwith no
motion measurement
https://doi.org/10.3389/fphys.2015.00147
Deblurringretinal opticalcoherence
tomography viaaconvolutionalneuralnetwork
withanisotropic anddoubleconvolution layer
Lianetal.(2018)
http://dx.doi.org/10.1049/iet-cvi.2018.0016
Blurred OCTimage (out-of-focus)
We resolve both out-of-focus and motion deblurring
inretinal OCTimagerywithin aunified framework.

77. SimilarlyyoucouldmeasureECG/PPGsimultaneously,andinvasiveBPwith
non-invasiveas “goldstandard”/low-qualitytrainingpairs
RobustHeartbeatDetectionfromMultimodalData viaCNN-basedGeneralizableInformation
Fusion BS Chandra, CS Sastry,S Jana (2018)https://arxiv.org/abs/1807.03232

78. DeepCompressedSensing forhigherframerates#1
Compressedsensinglaserscanningmicroscopy(2016)
BN.PAVILLON* AND N. I. SMITHBiophotonicsLaboratory, ImmunologyFrontier Research Center (IFReC),Osaka University
https://arxiv.org/abs/1807.03232

79. DeepCompressedSensing forhigherframerates#2
AVersatileCompressedSensingSchemeForFasterAnd LessPhototoxic3D
FluorescenceMicroscopy
MaximeWoringer, XavierDarzacq, ChristopheZimmer, Mustafa Mir
http://dx.doi.org/10.1364/OE.25.013668
We describe implementations on a lattice light sheet
microscope and an epifluorescence microscope, and show
that images of beads and biological samples can be
reconstructed with a 5-10 fold reduction of light
exposureandacquisitiontime.

80. Super-resolution for“thinkingmicroscopy”

81. Ifyoucando “videorate”microscopy,youcantrainamultiframesuper-
resolutionnetworkaswell(i.e.oversampleforimagerestoration)
Frame-RecurrentVideoSuper-Resolution
Mehdi S.M.Sajjadi, RavitejaVemulapalli,MatthewBrown (2018)
https://arxiv.org/abs/1801.04590
Invivoquantitationofinjectedcirculatingtumorcellsfromgreat
saphenousveinbasedonvideo-rateconfocalmicroscopy
HowonSeo,YoonhaHwang, Kibaek Choe, and Pilhan Kim (2018)https://doi.org/10.1364/BOE.6.002158
Biomedical OpticsExpressVol. 6, Issue6, pp.2158-2167 (2015)
Schematicofintravital custom-builtvideo-rate(30Hz) laser scanning
confocalmicroscopesystemandillustrationofgreatsaphenousvein(GSV)in
themouseleg:ND,neutraldensityfiler;DBS,dichroicbeamsplitter;BPF,band
passfilter;M,mirror;PMT,photomultipliertube; OBJ,objectivelens.

82. Measurethe PSFwithAdaptiveOptics andlearntheinverseproblem
Characterizationandadaptiveopticalcorrectionofaberrationsduringinvivoimagingin
themousecortex NaJi, Takashi R. Sato, and Eric Betzig (2012)https://doi.org/10.1073/pnas.1109202108
Ji et al. (2012): Lateral and axial images of GFP-expressing
dendritic processes (mouse cortex, 2-PM, 170 μm
Adaptiveopticsinmultiphotonmicroscopy:comparisonoftwo,
threeand fourphotonfluorescence David Sinefeld etal.(2015)
https://doi.org/10.1364/OE.23.031472
Phase correction for a 2-m-focal length cylindrical lens for 2-, 3- and 4- photon
excited fluorescence of Alexa Fluor 790, Sulforhodamine 101 and Fluorescein.
(a) Left – 4-photon fluorescence convergence curve showing a signal
improvement factor of × 320. Right – final phase applied on the SLM (b) left – 3-
photon fluorescence convergence curve showing a signal improvement factor of
× 40. Right – final phase applied on the SLM. (c) Left – 2-photon fluorescence
convergence curve showing a signal improvement factor of × 2.1. Right – final
phaseappliedon theSLM.Color-barsarein wavelengthunitscale.

83. SensorlessAdaptiveOptics simplifyinghardwarewithalgorithms
Coherence-GatedSensorlessAdaptiveOpticsMultiphoton
RetinalImagingMichelleCua,Daniel J. Wahl, YuanZhao,SujinLee, Stefano Bonora,RobertJ.
Zawadzki, Yifan Jian &Marinko V. Sarunic ScientificReportsvolume6,Articlenumber: 32223(2016)
https://doi.org/10.1038/srep32223
Optical Coherence Tomography (OCT) (top row)
and two-photon excited fluorescence (TPEF)
(bottom row, middle and right) images of the
mouse retina before and after OCT-guided
aberration correction.

84. AdaptiveOptics makescalciumelectrophysiologybettertoo
Calciumtransientsevoked bythestimulationof adrifting grating, 400 and 500mmbelowpia in theprimary
visual cortex of amouse(Thy1-GCaMP6slineGP4.3)

85. Deeplearningfor SpatialLightModulators aswell
Lightscatteringcontrolwithneuralnetworksin
transmissionandreflection
AlexTurpin, Ivan Vishniakou,and JohannesD.Seelig (2018)
https://arxiv.org/abs/1805.05602
Spatial light modulator(SLM, DMD, 768 × 1024 pixels, pixel size = 13.7 µm2
model V-7000 from Vialux)) at a maximum frame rate of 22.7 kHz

86. Deeplearningfor coded-illumination sourcedesign
Physics-basedLearnedDesign:Optimized Coded-
IlluminationforQuantitativePhaseImaging
Michael R. Kellman, EmrahBostan,NicoleRepina, Michael Lustig,Laura Waller
https://arxiv.org/abs/1808.03571
Learning Coded-Illumination Design for Quantitative
Phase Imaging: (a) Schematic of the LED-illumination
microscope where multiple intensity measurements are captured
under unique coded-illumination patterns, (b) Computational
phase reconstruction of the sample’s optical phase with coded-
illumination measurements. (c) Optimization for learning of coded-
illuminationdesignbasedonthenon-linear iterativereconstruction.
InvitroQuantitativePhaseImaging (QPI)
enablesthestain-andlabel-free imagingof
transparentbiologicalsamples.
Thisveryrelevantwhen
tryingtoimageretinal
ganglioncells
EthanA.Rossi etal.(2017)
10.1073/pnas.1613445114

87. AdaptiveOptics withthreephotons ratherthantwo
Sinefeld D,PaudelHP,WangT,WangM,OuzounovDG,Bifano TG,XuC: Nonlinearadaptive
optics:aberrationcorrectioninthreephotonfluorescencemicroscopyformouse
brainimaging.Proc SPIE2017 https://doi.org/10.1117/12.2252686
Here, we present a 3PM AO microscopy system
for brain imaging. Soliton self-frequency shift is
used to create a femtosecond source at 1675
nm and a microelectromechanical (MEMS) SLM
serves as the wavefront shaping device. We
perturb the 1020 segment SLM using a modified
nonlinear version of three-point phase shifting
interferometry. The nonlinearity of the
fluorescence signal used for feedback ensures
that the signal is increasing when the spot size
decreases, allowing compensation of phase
errors in an iterative optimization process without
direct phase measurement. We compare the
performance for different orders of nonlinear
feedback, showing an exponential growth in
signal improvement as the nonlinear order
increases. We demonstrate the impact of the
method by applying the 3PM AO system for in-
vivo mouse brain imaging, showing
improvement in signal at 1-mm depth inside
the brain.
SECTIONING
Horton et al. (2013)
Three-photon microscopy
2PM, attenuation
z2
from focal plane
3PM, attenuation
z4
from focal plane
osa-opn.org, November 2013
3-PM 601um 2-PM 429 um
Wang et al.
(2015)

88. Deeplearningtakingoverthe roleofphysicalcomponents aswell
EranHershko,Lucien E.Weiss,TomerMichaeli, YoavShechtman. Technion(2018)
Multicolor localizationmicroscopy bydeeplearning. ProcSPIE2017
https://arxiv.org/abs/1807.01637
First, we experimentally demonstrate an algorithm for
determining an emitter’s color using a standard fluorescence
microscope equipped with a grayscale camera with no
additional hardware modification. This is enabled by the fact that
the PSF of any optical system is dependent on the
wavelength, even without PSF engineering. Second, we
developandexperimentallydemonstrate anadditionalneuralnet
that algorithmically optimizes a color-encoding PSF using
phasemodulation,for maximalcolor-distinguishability.
To test whether a neural net could discriminate between two
types of emitters, we prepared a thin sample containing green
and red quantum dots (Qdots) with emission peaks at 565
and705nm,andimageditusing anepifluorescencemicroscope.
Here, we have demonstrated how deep learning is capable of
performing roles traditionally accomplished with physical
components. Post-process, software tools can be
advantageous over hardware-based methods due to a lower
implementation cost, system adaptability, and further
optimization without the requirement of collecting new,
experimentaldatasets.
To optimally discriminate between PSFs, we have shown that
PSF-engineering can be done in coordination with net training
to maximize on the strengths of the reconstruction net, which do
notfollowthesameprocessasmost-likelihoodestimators.

89. Marryingdeeplearning withMonteCarlophysics-basedmodelling
AnalyzingInverseProblemswithInvertibleNeuralNetworks
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert,Daniel Rahner, EricW.Pellegrini, RalfS. Klessen, LenaMaier-Hein,Carsten Rother, Ullrich Köthe (Submitted on 14Aug 2018)
https://arxiv.org/abs/1808.04730
The results produced by the INN provide several new insights: First, we find that
the posteriors for layer thickness and anisotropy match the shape of their
priors, i.e. y holdsno information about these parameters– theyare unrecoverable.
Second, we find that the sampled distributions for the blood volume fraction and
scattering amplitude are strongly correlated. As blood volume fraction
increases, more light is absorbed inside the tissue. For the sensor to record
the same intensities y asbefore, scattering must be increasedaccordingly.
While the correspondence between simulations and real measurements remains to
be established, we share the excitement of the application experts to push INNs
towards a generic tool, helping scientists from many different disciplines to better
interpret theirdata and models, and to better plan their next experimental steps
– be it modeling,measuringorsimulation
In medical science, the functional state of biological tissue is
of interest for many applications, such as tumor detection or
verifying organ transplantation success. Tumors, for
example, are expected to exhibit changes in oxygen
saturation. Such changes influence the reflectance of
the tissue, which can be measured by multispectral
cameras.
We can simulate these measurements from a tissue model
involving oxygen saturation, blood volume fraction,
scattering magnitude, anisotropy, and tissue layer thickness
[Wirkertetal.2016]. While these simulations can determine
the reflectance spectrum (y) for a given tissue, inverting the
measurements to recover the underlying functional
properties(x) isanactivefieldofresearch.
We train an INN for this problem, along with two ablations
(only forward or only inverse training), as well as a regular
neural net using the method of Kendall andGal 2007, with
Monte-Carlo (MC) dropout and additional aleatoric error
termsforeach parameter.

90. TheMoreDataThemoreyoucansynthesizedataaswell
Fast3Dcelltrackingwithwide-fieldfluorescencemicroscopythroughdeeplearning
KanLiu, Hui Qiao, Jiamin Wu, Haoqian Wang, LuFang QionghaiDai (2018)
https://arxiv.org/abs/1805.05139
Framework of the proposed 3D localization microscopy. Lateral detection CNN, highlighted by the blue
dashed box, first determines whether there exist diffraction patterns at the central lateral position of the sliding window.
Axial localization CNN, highlighted by the orange dashed box, then estimates the axial positions of the predicted
positive samplesof lateral detection CNN.
Therefore, 3D positions of the fluorescent probes are finally acquired. Making use of the determined 3D localization
results, fast 3Dtracking can be realized with aKalmanfilter.
The large amounts of training data for our framework is obtained from the simulation of the
incoherent superposition of multiple objects with the prior knowledge of the z-stack of a single object.
While thez-stack oftheobjectcan be synthesized bythesimulated pointspread function (PSF)and the
shape of the object, we choose an experimental z-stack of a single object for training data synthesis
due to the diverse imaging environments in different experiments, such as the optical aberration, medium
inducedrefractiveindex mismatch, andnoisecondition.
Tracking blood cells (75 µm/s) at 100 fps of a one-day-old
live zebrafish restrained in agarose. (a) Captured wide-field
fluorescence images of the ROI at different time stamps and the
corresponding localization results reconstructed by our method
and MLE method. (b) 3D tracking results of the blood cells
reconstructedbyour methodandMLEmethod.

91. Manyfieldsofopticsbenefitfromlargedatasetstobeusedfororwithsynthesis
pipelines.Deeplearningfor ultrashortpulsereconstruction
Deeplearningreconstructionof ultrashortpulsesTom
Zahavy,AlexDikopoltsev,Daniel Moss, Gil Ilan Haham,Oren Cohen, ShieMannor, and
Mordechai SegevOpticaVol. 5,Issue5,pp. 666-673 (2018)
https://doi.org/10.1364/OPTICA.5.000666
Here, we propose and
demonstrate, theoretically and
experimentally, the reconstruction
of ultrashortoptical pulsesby
employingdeepneural networks
(DNNs),andshow (on simulated
data) that ourtrainednetwork
outperformsother state-of-the-art
techniquesfor low SNR
measurements.
We furtherdevelop our
methodology bymodifyingthe
network trainingstage to combine
bothsupervisedandunsupervised
learning, and showthat thisnew
network isable to reconstruct
ultrashort pulsesfrom low SNR
experimental data, while being
trained on simulated data.
To further enhance the performance of our
approach, we plan to investigate the sim-to-real
challenges in future work. First is increasing the
variety of the computer-generated dataset to include
asmany spectral amplitudesand phasesaspossible.
The second is to significantly enlarge the number of
measured pulses that train the network. Of
course, this suggestion has obvious disadvantages, but
in some experimental schemes, where the
measurements are embedded in noise, or when extreme
accuraciesare crucial,thiscould be practical.
The third is using generative models to generate
more data by learning the data distribution of measured
pulses. In particular, a recently developed network called
the generative adversarial network (GAN) [34]
can be used to create new data pulses on which the
DNN tends to make mistakes (poorly reconstruct the
pulses). These pulses will be new to the dataset on
purpose, and will increase the variety of the pulses in the
trainingdataset.

92. Synthesizingnewsamples asweknowprettywellwithallthetweaksthe
“imagemodel”/latentspace,creating “virtualanimals”
RobustHeartbeatDetectionfromMultimodalData viaCNN-basedGeneralizableInformation
Fusion BS Chandra, CS Sastry,S Jana (2018)https://arxiv.org/abs/1807.03232
Virtual patient generation with possibly different
paravalvular leakage (PVL) levels, for patients with transcatheter
aorticvalvereplacement(TAVR)
Combining a convolutional neural network (CNN) and
generative adversarial networks (GAN), we discover
the pathophysiologic meaning of the feature
space. This demonstrates l generative invertible
networks (GIN) can generate virtual patients not only
visually authentic but also pathophysiologically
interpretable