Multispectral Purkinje Imaging
•
2 likes•494 views
Developing an inexpensive optical measurement device to estimate ocular media density. Selection of HDR camera, deep learning software stack for Purkinje image detection and segmentation, NVIDIA Jetson as the embedded computer triggering either LED or LASER lights to be projected on the cornea Alternative download link: https://www.dropbox.com/s/fh7r8szuc2pctfr/purkinje_imaging_inPractice.pdf?dl=0
Recommended
Recommended
More Related Content
What's hot
What's hot (15)
Similar to Multispectral Purkinje Imaging
Similar to Multispectral Purkinje Imaging (20)
More from PetteriTeikariPhD
More from PetteriTeikariPhD (14)
Recently uploaded
Recently uploaded (20)
Multispectral Purkinje Imaging
- 1. Petteri Teikari, PhD Singapore Eye Research Institute (SERI) Visual Neurosciences group http://petteri-teikari.com/ Version “Thu 6 September 2018“ Multispectral Purkinje Imaging developing an inexpensive measurementdeviceto estimateocular media density 1938CarlMydans
- 4. Purkinje Images Image Processing Rextractautomatically from the HDR frame(s) the diference insignal intensity (region proposal / localization Fourth image isformed after adouble passthrough the crystalline lens 1) REGIONPROPOSAL https://medium.com/@smallfshbigsea/faster -r-cnn-explained-864d4fb7e3f8 2) Pixel-by-pixel segmentation, a.k.a. “SemanticSegmentation” https://github.com/GeorgeSeif/Semantic-Segmentation-Suite The raw A/D output was then converted to a radiance profle using calibration data for the camera/frame-grabber system. Custom software allowed automated analysis of the averaged radiance profles for each interference flter. A least squares linear regression line was ftted to the flanns of each radiance profle (representing bacn-scattered light from various layers in the crystalline lens, etc.), and the radiance for the point on the regression line lying directly underneath the peak of the IVth image was calculated. The radiance for this position on this downward sloping scatter shoulder was subtracted from the gross radiance at the pean of the radiance profle. Thus, the net reflected radiance of the IVth Purninje image alone, or output, was computed separately for eachwavelength. The subsequent calculation of individual lens density consisted of taning the common log of the ratio of the input radiance to the net refected output radiance of the IVth image. The result was divided by two to compensate for the double-pass through the anterior segment of the eye. Several such density measurements were made, and the average density was calculated for each eye of every subject. No adjustment was made for the path length diference (about 3%) caused bythe beam beingan average of about 15° of-axis or forpupillary area.
- 5. Applications Ocular Media Density van deKraatsandvan Norren2007 Teinari and Najjaret al. 2012 Cataractassessment deCastro etal.2018 DohyeunKim et al. 2018 Intraocular lens(IOL) issues CRSToday deCastroetal.2007 Schaefel2008 Nishi etal.2010 Sun et al.2014 Maedeletal.2017 Kumaretal.2018 +Laboratorio deOptica,UniversidaddeMurcia + InstitutodeÓptica Daza deValdés,Consejo Superior deInvestigacionesCientífcas,Madrid +Instituteof Ophthalmology,University College London,andMoorfeldsEyeHospital NHS Foundation Trust +Vienna Institutefor ResearchinOcular Surgery +Institutefor OphthalmicResearch,Tubingen (a) Three-dimensional (3-D) optical coherence tomography (OCT) image (example is for S#3, OD); (b) en face OCT image showing the pupil merging, the locations of PI, PIII,and PIV in red and pupilcenter in yellow; (c) PI (left), PIII (center), and PIV (right) in OCT B-Scan images. -Sun et al.2014 Kumaretal. 2018 The clinical feasibility of the Spanish Purkinje meter system was better than that of the German Purninjemeter. In pseudophanic eyes, the German Purninje meter measured signifcantly higher IOL tilt and decentration values than the Spanish Purninje meter.
- 6. Imaging
- 7. Camera Multiframe HDRconstruction or/and HDRcamera The faster the framerate, lessthere will be motion artifactsbetweensuccessive frames (eachwith diferent wavelength) e-CAM20_CU0230_MOD (NVIDIAJetson TX2/TX1) See3CAM_CU20 HDRUSB3.1 Gen1 CameraBoard (Color) CMOSsensor 12-Bit RGB Bayer Dynamic Range:Max upto105dB SensorStrobeoutputto connectexternal flash ElectronicRolling Shutter Fixed Focuswith M12lensholder VOLUME PRICE STARTFROM US$59-89 Petteri: Monochrome camera withouttheBayer arraywould bebetterfor thisproject Ximea MQ003MG-CM CMOSISCMV300 https://www.ximea.com/en/products/usb3-vi sion-cameras-xiq-line/mq003mg-cm https://www.ximea.com/downloads/usb3/manuals/xiq_technical_manual.pdf Digitizationwith12bits HDR:KneepointbasedHDR Mane your life easier with a proper camera, and when you have the idea worning tryto optimize the cost→ C-mount [or with C/CSmount moduleB]
- 8. HDR ”Kneepoint basedHDR” Asprovided by Ximea maning your initial eforts easier https://www.ximea.com/down loads/usb3/manuals/xiq_tech nical_manual.pdf
- 9. Software
- 10. Ximea Support SupportedVisionLibrariesFor anup-to-datelistingofthesupportedvisionlibrariesandsoftwarepacnages http://www.ximea.com/support/projects/vision-libraries/wini National Instruments LabVIEW Vision Library MVTecHALCONMathWornsMATLAB ImageAcquisitionToolbox GOFORTHIS OpenCV is an open source library of programming functions mainly aimed at real time computer vision, developed by Intel and now supported by Willow Garage. More: http://opencv.willowgarage.com/wini/ https://www.ximea.com/support/wini/vision-libraries/OpenCV
- 11. OpenCV Comingwithmany of-the-shelfdeep learningalgorithms already CASE: objectdetection andtracking object= purninje reflection(foreach object) OpenCVObject Tracking ByAdrianRosebrocnonJuly30,2018 JoeyAdrianRosebrocn inObjectTracning,Tutorials https://www.pyimagesearch.com/2018/07/30/opencv-object- tracning/ MultiTracker:MultipleObjectTracking usingOpenCV(C++/Python) AUGUST 5,2018 BY SATYAMALLICK https://www.learnopencv.com/multitracner-multi ple-object-tracning-using-opencv-c-python/ Before we dive into the details, please checn previous posts listed below on Object Tracning to understand the basics of single object tracners implementedinOpenCV. ObjectTracningusingOpenCV GOTURN:DeepLearningbasedobjecttracning
- 12. Semantic Segmentation 4th PurkinjeImage 1st PurkinjeImage 3rd Purkinjeimage http://pabloartal.blogspot.sg/2009/02/eyes-are-not-center ed-optical-systems.html Ocularexampleof SemanticClasses Pupil Purkinjerefections If you have irises and eyelashes annotated, the previous step of detection might become even easier (i.e. pupil found inside iris, and reflections inside pupil, etc.)
- 13. Semantic Segmentation Implementationin OpenCV SemanticsegmentationwithOpenCV anddeeplearning by AdrianRosebrocn on September 3,2018 in DeepLearning, SemanticSegmentation, Tutorials https://www.pyimagesearch.com/2018/09/03/semantic-segmentation-with-opencv-and-deep-learning/ Inthistutorial,youwilllearnhowtoperformsemanticsegmentationusingOpenCV,deeplearning,andthe ENetarchitecture(Paszneetal.2016:ENet: ADeepNeuralNetworn Architecturefor Real-Time SemanticSegmentation,Citedby163).Afterreadingtoday’sguide,youwillbeabletoapplysemantic segmentationtoimagesandvideousingOpenCV. https://medium.com/@robbertbormans/summary-of-enet- a-deep-neural-networn-architecture-for-real-time-semantic -segmentation-300a24f31e43 https://www.youtube.com/watch?v=1iK1gMOVZF0
- 14. Tracking+ Semantic Segmentation Computingdetectionand segmentation jointly should in theorygive youeven betterresults Imagesegmentation withMask R-CNN JonathanHui DeepLearning https://medium.com/@jonathan_hui/i mage-segmentation-with-masn-r-cnn -ebe6d793272 Stepbysteptutorialonrunninga MaskR-CNN with OpenCV UsingMaskR-CNNinthestreetsof BuenosAires –NicolásMetallo
- 16. Plentyofoptions outthere Annotatethe regionsof interestwithsomenice open-source annotation “engine” line LabelMe Donothaveapoor studentdrawing outlines onPhotoshop/ GIMP /MS Paint:P wnentaro /labelme ImagePolygonalAnnotationwithPython (polygon,rectangle,line,pointandimage-levelflagannotation) Labelmeisagraphicalimageannotationtoolinspiredby http://labelme.csail.mit.edu. https://github.com/topics/image-annotation ?l=python
- 17. Leverageexisting eyedatasets Pooltogetherorfne-tune (transferlearning, domain adaptation)the existing detection /segmentation networns Tonsen, M.,Zhang,X., Sugano, Y., Bulling,A., 2016. Labelledpupilsinthewild: adataset forstudyingpupildetectionin unconstrainedenvironments,in: Proceedings of theNinth Biennial ACM Symposiumon EyeTracning Research &Applications, ACM.pp. 139–142. https://arxiv.org/abs/1511.05768 → Citedby19 Wepresentlabelledpupilsinthewild(LPW),a noveldatasetof66high-quality,high-speedeye regionvideosforthedevelopmentandevaluationof pupildetectionalgorithms. https://www.mpi-inf.mpg.de/departments/computer-v ision-and-multimodal-computing/research/gaze-bas ed-human-computer-interaction/labelled-pupils-in-th e-wild-lpw/e.g. Anovelapproachtosinglecamera,glint-free3D eyemodelfttingincludingcornealrefraction KDiernes,M Kassner, ABulling annotated pupilcenters [Fuhl etal. 2015, ExCuSe]
- 18. Triggering
- 19. Ximea Triggering https://www.ximea.com/downloads/usb3/manuals/xiq_technical_manual.pdf xiQmodelXimeaMQ003CG-CMMQ003MG-CM HWTrigger Trigger withoverlap(see4.3.2.2Triggeredmodewithoverlap) Several sensors are capable to trigger exposure in overlap mode, so it is capable to reach the same frame rate as in free run mode. When the trigger period is longer than the exposure and readout time, the signal wave form will loon similar to Triggered mode without overlap. However when the trigger period is decreased, the sensor will expose the images in overlap mode. In this case, the frame active signal will be constantlyactive.
- 20. Ximea MQ003MG-CM 500fpsatVGA - 500 spectral bandsinasecond? More line lessbandsreally fast toreduce ocular motion during acquisition withHDR mode. Ifyou automate thiswell,theacquisition+ processing couldbeiterated duringone secondmultiple timestomanesurethatthe qualityisgood,andyou do nothaveto realizeduring ofineanalysis(avoidthis)that you wouldneedtore-measurethatperson. Ximeasupportsthefollowing exposuretimes: Exposuretime(EXP) 54µsto1sec instepsof7.56µs Soifyoucannothave brightenough illumination for54µs,youcanexpose longer (orthatbrightness isabove thesafetylimits) Thesedependon yourilluminants (intensity and wavelength) For example 400 nm would need to be 2x brigher than 500 nm with the same exposure time
- 22. Eventuallyyou wanttodeploy thisasportable device Mightaswell have the I/O handled bysome embedded device? GofortheTX2 DeveloperKit! RaspberryPiZeroW low-costbut willnotbepowerfulenough forimage processing JetsonTX1 isanembeddedmodulefromNvidia whichincludesquad-core64 bitARMCPUand256- coreMaxwellGPU. Itdelivershighperformanceandpower efciencyin smallsizeandcanbeusedfor deeplearning, computer visionandother computing applications. JetsonTX2 isasuccessor toTX1,updatingGPU architecturetoPascalandimprovingtheperformance.
- 23. Embedded CameraSystem Example with OpenCV, NVIDIATX2and USB Camera https://devtaln.nvidia.com/default/topic/1017860/jetson-tx2/usi ng-on-board-tx2-camera-with-opencv-3-2-/ https://devtaln.nvidia.com/default/topic/1036193/jetso n-tx2/tane-pictures-with-csi-camera-on-tx2/ Community exists KirkKaiser ArtistandSoftwareDeveloper ManeArtwithPython IusePython3,along withthe OpenCV3,andDarnflow,asIdid inPoor Man’sDeep Learning Camera post Youcanusemy forn ofthe JetsonHacns’ ofan OpenCV3buildscript.
- 24. Illumination
- 25. Ifyoumoney, and fast switching times with narrow bands gowithNKTFianium WhiteLase Class3B supercontinuumlaser and AOTF flters (Goochand Housego) https://www.nntphotonics.com/lasers-fbers/product/fanium-whitelase-micro/ https://goochandhousego.com/product-categories/tunable-flters-aotf/
- 26. Back toReal World Easiertouse quasi- monochromatic LEDs Use more bandson shortwavelenghts asthe ocularmedia variesthere more between subjects OcularAbsorbers Function ofage 80yr 20yr Opto-electronics > LEDLighting > LEDEmitters https://www.mouser.sg/Optoelectronics/LED-Lighting/LED-Emitters/_/N-8usfd Select as many wavelengths that you can “comfortably” use and ft the template provided by van deKraatsandvan Norren (2007) for your measured data to estimate the full spectral characteristics of theocular media. van deKraatsandvan Norren (2007)
- 27. Orwith inexpensive solid-state lasermodules Monochromatic light sources,unlessyou can aford touse interference flterswith yourLEDs HackingCD/DVD/Blu-rayforBiosensing(2018) http://doi.org/10.1021/acssensors.8b00340 Theoptical picnupunit(OPU)within aCD/DVD/Blu-raydriveintegrates 780,650,and 405 nm wavelength lasers,difraction- limitedoptics,ahigh-bandwidth optoelectronictransducerupto 400MHz,andananoresolutionx-, z-axis,andtiltactuatorinacompact size. Osram laserdiode PLT5450B 80mW, 450nm 33SGD/unit Osram laserdiode PLT5520B 80mW, 520 nm 55SGD/unit Alibabalaserdiode 410-420nm Price? RPMC Lasers TO-56 50mW,430 nm Price?
- 28. Ormakeyour colorsfrom singlelaser lightsource? 430nmlaserpumped polychromaticlight Polychromaticliquidcrystallaser arraystowardsdisplayapplications Stephen M. Morris, Philip J. W. Hands, Sonja Findeisen-Tandel, RobertH. Cole, TimothyD. Wilninson, and HarryJ. Coles https://doi.org/10.1364/OE.16.018827 → Cited by 61 Continuouswavelength-tuningacrossthevisiblespectrumwith a compactandinexpensive liquidcrystallaser PJW Hands, SM Morris, MM Qasim, DJ Gardiner,TD Wilninson, HJ Coles Departmentof Engineering, University ofCambridge, UK 24th International Liquid Crystal Conference (ILCC2012), Mainz, Germany
- 29. Optics
- 30. Combine yourlaser beamsonsame opticalpathtobe projectedoncornea http://www.fberlaser.fujinura.jp/eng/products/ about-fber-laser.html Powerenhancement of fberlasers Figureshowshowpowerisenhanced by couplingthemultipledeliveryfbersshownin Fig. 3withanoutput beamcoupler(combiner). Forexample,couplingfour1nW outputbeams yields4nW,andcouplingsix 1nW output beamsyields6nW. RGB 200mWWhite Laser Module/TTLRGB Laser/ US $53.00 520nm(30mW)+638nm(100mW)+445nm(80mW) BeamCharacteristics:(Nearto TEM))model ,Divergence<3.5mrad, Spotroundness>85% Buy aChineseRGBlaser module and trycombine that withsome short-wavelength lasers? Thorlabs LaserDiodeTutorial
- 31. RGBLED ocombinationwith of-the-shelfoptics https://www.youtub e.com/watch?v=l-F B6tQbpQM underthe microscope, how doesastrip of LEDs worn? https://sg.rs-online.com/web/p/led-lenses/8892245/
- 32. Shapethe light Tohavethearc- shapedpattern (orsimilarpatternthat iseasy todetect whenflipped betweenthe4th and3rd Purninjeimage) Maybedifuse toremovethe shapesofindividualLEDs? 4th PurkinjeImage 1st PurkinjeImage 3rd Purkinjeimage https://www.luminitco.com/products/light-shaping-difusers Variouswaystoshapeyourlightbeam Half-toneproximitylithography (2010) http://doi.org/10.1117/12.854307 →Citedby27
- 33. Orsimple GOBO pattern? Tohavethearc- shapedpattern GoboCatalog| Rosco orderinga customgobo-Rosco Gobos- Apollo Design GoboandGAMPatternsearch PatternGobo (s)LaserUnits Remember! Thispattern shouldbedesignedso thatis: 1) Easy to “fnd”bythe MaskR- CNN fromthecornea. 2) AndtheMasn R-CNNeasily fndsthe invertedimage and candistinguish 3rd and4th Purninje imagesfromeach other