Hand Pose Tracking for Clinical Applications

Hand Pose
Tracking
Clinical applications
withafocuson
rheumatoidarthritis(RA)
management
Version “25/07/20“
Petteri Teikari, PhD
MScElectricalEngineering/
PhDNeuroscience
https://www.linkedin.com/in/petteriteikari/ https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/Input/HandTracking.html

Hand
Tracking
Wherehand
andallits
jointsarein
timeandspace

Where doeshand pose matter? #1
Thebonesoftheforearm,wrist,andhand[Bla14].Acronyms:CMC
–Carpometacarpaljoint,MCP–Metacarpophalangealjoint,PIP–
Proximalinterphalangealjoint,DIP–Distalinterphalangealjoint
Blausen.com staff (2014). "Medical galleryof Blausen Medical 2014". WikiJournal
ofMedicine 1 (2). DOI:10.15347/wjm/2014.010. ISSN 2002-4436.
https://doi.org/10.3389/fcomp.2019.00008

https://venturebeat.com/2020/07/20/researchers-show-fingertrak-a
-hand-tracking-wristband-for-ar-vr-input/
https://docs.microsoft.com/en-us/windows/mixed-rea
lity/unreal-hand-tracking
https://www.youtube.com/watch?v=5oojAiryxwk
https://www.roadtovr.com/oculus-quest-ha
nd-tracking-hands-on-oculus-connect-6/
Getthe handjoint 3Dcoordinatesasaccurately
aspossibleforclinical applications
Airealisahapticstechnologyco-developedby
Rajinder SodhifromtheUniversityofIllinoisandIvan
PoupyrevfromDisneyResearch.
https://virtualrealitytimes.com/2017/02/15/list-of-hand-tracking-tech-vr/
https://www.theverge.com/2019/5/30/18645604/leap-motion-vr-hand-tracking-ultrahapti
cs-acquisition-rumor

The biggest driverforhand tracking isVRnowactually
→ Betterinteraction withthe virtualworld
According to Marketsandmarkets ,
the market for gesture recognition is
expected to grow from USD 9.6
Billion in 2020 to USD 32.3 Billion
in just 5 years. More and more
companies are adopting the
technology to ease the lives of their
customers and solve some of their
everyday problems. This is a win-win
situation.
https://virsabi.com/the-future-of-vr-is-in-your-hands/
#handtracking #OculusQuestInterhaptics
https://www.facebook.com/watch/?v=605635140389695
24July2020

HandPose
Datasets,
Code
andSDKs

Hand Pose – DatasetsandImplementations
https://github.com/xinghaochen/awesome-
hand-pose-estimation
e.g.https://github.com/bardiadoosti/HOPE

Hand Tracking– API/SDK #1especiallyusefulforVR/AR
On-Device,Real-Time HandTracking
withMediaPipe
Monday, August19, 2019
ValentinBazarevsky andFanZhang,
Research Engineers,GoogleResearch
weemploy TFLiteGPUinference onmostmodernphones

Hand Tracking– API/SDK #2 especiallyusefulforVR/AR
https://developer.apple.com/videos/play/ww
dc2020/10653/
https://www.intelrealsense.com/hand-tracking-overview/
https://github.com/ReiiYuki/Intel-Realsense-Hand-Toolkit-Unity
https://docs.microsoft.com/en-us/azure/Ki
nect-dk/body-sdk-download
https://www.youtube.com/watch?v=gpQePH-Ffbw
https://www.manomotion.com/https://hub.vive.com/storage
/tracking/overview/feature.ht
ml
https://youtu.be/yV1-N58GiOo

Hand Tracking– API/SDK #3
https://developer.leapmotion.com/documentation
SuggestedVRProjects
Strokerehabilitation. Encouragestrokepatientstoachievetheir
treatmentgoalswithentertaininggameplaymechanics.
Industrialtraining. Createavirtualflightcockpitor guideusersthrougha
complicatedprocedure.
Education. Guideusersthroughaninteractivedatavisualizationthat
teachesthemacomplexconcept.
https://newatlas.com/manomotion-hand-tracking-sdk-smartphones/49811/
https://www.intelrealsense.com/types-of-tracking-overview/

Hand Tracking– API/SDK forfuturegaming
MAY13,2020
Sonydesigningfinger-trackingVRcontroller
byPeterGrad,TechXplore
https://techxplore.com/news/2020-05-sony-finger-tracking-vr.html
#VR #PSVR2 #VRControllers
Prototype PlayStation Next-Gen VR ControllersWith
Finger-Tracking!
183,669 views•May3, 2020
https://youtu.be/IZpgjymt9Xc
Evaluationof MachineLearningTechniques for Hand
PoseEstimationonHandheldDevice withProximity
SensorKazuyukiArimatsuandHideki Mori Sony InteractiveEntertainment Inc.,Minato-ku,
Japan
CHI'20:Proceedingsofthe2020CHIConferenceonHumanFactorsin
ComputingSystemsApril2020Pages1–13
https://doi.org/10.1145/3313831.3376712
"We aim to push the boundary of finger-
tracking capability for more intuitive
interaction using hands in virtual space,"
Arimatsu and More said. "Our approach not
only detects touching of fingers on a specific
electrode but predicts comprehensive finger
movement in 3D space utilizing values from all
sensors. To achieve that, we evaluated two
types of convolutional neural network
architectures studied in the computer vision
field for pose estimation, and illustrated the
suitable architecture for the sensors on the
controller."

Hand Tracking– API/SDK for“socialgaming”
EventhoughPeter Chouhadlong partedways withtheVRbusinesshecreatedatHTC,
theformer CEOcontinuedtowork intheindustry,whichhasincludedforaysintoa
visualeffectsstudio anda VRgamestudio.Butwhattheexeckepttight-lippedaboutover
thepastthreeyearswasaprojectcalled XRSpace,astartupthatclaimsitwill
revolutionizethecurrent“gimmicky,”“lonely”and“complicated”VRexperience.Theresult
isanewVRheadsetaimedatregular consumerswithintuitivenessandasocial-first
approach.
https://www.engadget.com/xrspace-mova-5g-vr-headset-hand-tracking-space-scanning-083936618.html

Hand Tracking– API/SDK smartphone gestures
notexactly0.1° jointangle accuracy, butgood enough todetectuser’shand gestures?
https://patent.nweon.com/9815
https://atap.google.com/soli/
https://doi.org/10.1145/
3027063.3053152

Hand/PoseTracking microwave sensing becoming athing
nice to see through walls and non-invasively measurevitals
XeThruX4M200
https://www.xethru.com/x4m200-respiration-sensor.html
BodyScan: Enabling radio-basedsensing on wearable devicesfor
contactlessactivity andvitalsign monitoring Biyi Fang‡, NicholasD. Lane†∗,MiZhang‡,
Aidan Boran†, FahimKawsar† ‡MichiganStateUniversity, †BellLabs, University College London∗University College London
MobiSys'16 Proceedingsof the 14th
Annual International Conference on Mobile Systems, Applications, and Services
https://doi.org/10.1145/2906388.2906411 Cited by 44
http://news.mit.edu/2018/artificial-intelligence-senses-people-through-walls-0612
Machine-learningreprogrammablemetasurfaceimager
Lianli Li et al.(2019)withmetamaterials https://doi.org/10.1038/s41467-019-09103-2 Cited by 70 - Related articles
Citedby→ Revolutionizing FutureHealthcareusingWirelessontheWalls(WoW)
https://arxiv.org/abs/2006.06479 byJalilurRehman Kazimetal. (2020)
“A proposed solution is theuseof intelligentreflectivesurfaces,whichwill havetheability to manipulateelectromagnetic(EM)
signals. Theseintelligentsurfacesmounted and/or coated on wallsaka- IntelligentWallsareplanar and activesurfaces,which willbea
keyelementin beyond 5Gand 6G communication.Theseintelligentwallsequipped with machinelearning algorithmand computation
powerwould havetheabilityto manipulateEM wavesand actasgatewaysin theheterogeneousnetwork environment. “

Hand Tracking in VR/AR/MRApplications
https://www.virtualrealitypulse.com/hand-tracking/

Ground
Truths
Againstwhat
youaretraining
yourmodels

Themostobvious multi-camerasetupforhands
https://doi.org/10.1115/1.4005695
Multi-camerasetupwithKinectAzures
https://docs.microsoft.com/en-us/azure/kinect-
dk/multi-camera-sync
High-end VICONsorsomethingmore
affordable
http://docs.ipisoft.com/User_Guide_for_Multiple_Dept
h_Sensors_Configuration
http://wiki.ipisoft.com/User_Guide_for_Dual_Dep
th_Sensor_Configuration_(printable_version)
Build alow-cost motion capture system
https://youtu.be/XIw9cZ8i4mE

GettingGroundTruths?
Nice tohave somegoodqualitydatasetswhere the jointlocationshaveverysmall errors havingan→
estimate ofthe ‘goodness’ ofyou computer vision-based approximation (+auto-annotationwithextra
hardware wouldniceforyourcustomdatasets)
Wearablesensors,ordatagloves,arepromisingforaccurateand
disturbance-freehandmodelingsincetheygenerallyhavecompactdesignand
becomelighterandlesscumbersomefor dexteroushandmovements.However,
therearethreemainchallengesremainingtobesolved.
Mostdataglovesarestill“inthelab” andthereisnoindustrialstandard on
thedesignandfabricationofsuchdevices,whichleadstohighcostsofavailable
commercialproducts,making themunaffordablefordailyuse.Second,except
glovesthatarebasedonstretchsensors,mostgloveshavefixedsizeandare
difficulttomatchdifferentusers’ hands.Lastly,glovesare unsuitabletobeused
incertaincases,forexample,somestrokepatientshavedifficultiesopening
their handstowearglovesdesignedfor normalusers,orinsituationswhenthe
user needstomanipulatetinyobjects,or puttheirhandsintowater,etc.
Vision-basedmethods,ontheotherhand,haveovercomemanydifficulties
facedbycommoncomputervisiontasks,suchasrotation,scaleandillumination
invariance,andclutteredbackgrounds.However,vision-basedmethodsstillface
thefollowingchallenges:
First,occlusionisstillthemajor problem.Asthehandsareextensivelyusedto
manipulateobjectsindailylife,theyareverylikelytobeblockedorpartially
blockedbyobjectsduring interaction,whichformsthehand–object–interaction
(HOI)problem.
Second,sincemanymethodsaredata-driven,thequalityandcoverageof
trainingdatasetsisofgreatimportance.However,a largerportionof
annotateddatacomesfromsyntheticsimulations.
ASurveyonHandPose EstimationwithWearable
SensorsandComputer-Vision-BasedMethods
WeiyaChenetal.2020https://doi.org/10.3390/s20041074

GettingGroundTruths?#2
HandSeg:AnAutomaticallyLabeledDataset forHand
SegmentationfromDepthImages
AbhishakeKumar Bojjaetal.(2019)
https://doi.org/10.1109/CRV.2019.00028
https://vision.uvic.ca/pubs/2019/bojja2019handseg/page.md
We propose an automatic method for generating high-quality
annotations for depth-based hand segmentation, and
introduce a large-scale hand segmentation dataset. Existing
datasets are typically limited to a single hand. By exploiting the
visual cues given by an RGBD sensor and a pair of colored
gloves, we automatically generate dense annotations for two
hand segmentation. This lowers the cost/complexity of
creating high quality datasets, and makes it easy to expand the
datasetinthe future.
OnlineOpticalMarker-basedHandTrackingwithDeep
Labels ShangchenHanetal.(2018)Facebook Reality Labs
https://doi.org/10.1145/3197517.3201399
Optical marker-based motion capture is the dominant way for
obtaining high-fidelity human body animation for special
effects, movies, and video games. However, motion capture has seen
limited application to the human hand due to the difficulty of
automatically identifying (or labeling) identical markers on self-similar
fingers. We propose atechnique thatframesthelabeling problemasa
keypoint regression problem conducive to a solution using
convolutional neural networks. We demonstrate robustness of our
labeling solution to occlusion, ghost markers, hand shape, and even
motionsinvolvingtwo handsor handheldobjects

GettingGroundTruths?#3 Coloredgloves
Real-Time Hand-TrackingwithaColor Glove
ACMSIGGRAPH2009RobertY.WangandJovanPopović
http://people.csail.mit.edu/rywang/handtracking/

Learningthesignaturesofthehuman
graspusingascalabletactileglove
Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu,
AntonioTorralba & Wojciech Matusik
Nature volume 569, pages698–702(2019)
https://doi.org/10.1038/s41586-019-1234-z
The sensor array (550 sensors) is assembled on a
knitted glove, and consists of a piezoresistive film
connected by anetwork ofconductive thread electrodes
that are passively probed. Using a low-cost (about
US$10) scalable tactile glove sensor array, we record a
large-scale tactile dataset with 135,000 frames, each
covering the full hand, while interacting with 26 different
objects.
Interactive HandPoseEstimationusingaStretch-SensingSoftGlove
OliverGlauseretal.(2019) https://doi.org/10.1145/3306346.3322957
We propose a stretch-sensing soft glove to interactively capture hand poses with
high accuracy and without requiring an external optical setup. We demonstrate how
our device can be fabricated and calibrated at low cost, using simple tools available in
most fabrication labs. To reconstruct the pose from the capacitive sensors
embedded in the glove, we propose a deep network architecture that exploits the
spatial layoutof the sensoritself.

Adeep-learnedskinsensordecodingthe epicentral
humanmotionsKyunKyu Kimetal.(2020)
NatureCommunicationsvolume11,Articlenumber:2149
https://doi.org/10.1038/s41467-020-16040-y
We introduce a new measuring system, a novel electronic skin
integrated with a deep neural network that captures dynamic
motions from a distance without creating a sensor network. The device
detects minute deformations from the unique laser-induced crack
structures. A single skin sensor decodes the complex motion of five
finger motions in real-time, and the rapid situation learning (RSL)
ensures stable operation regardless of its position on the wrist. The
sensor is also capable of extracting gait motions from pelvis. This
technology is expected to provide a turning point in health-monitoring,
motiontracking,andsoftrobotics.

Haptic-feedbacksmart gloveasacreative human-
machine interface(HMI) forvirtual/augmented
realityapplications Minglu Zhu etal.(2020)National University of
Singapore
http://doi.org/10.1126/sciadv.aaz8693
Human-machine interfaces (HMIs) experience increasing
requirements for intuitive and effective manipulation. Current
commercialized solutions of glove-based HMI are limited by either
detectable motions or the huge cost on fabrication, energy, and
computing power. We propose the haptic-feedback smart
glove with triboelectric-based finger bending sensors, palm
sliding sensor, and piezoelectric mechanical stimulators. The
detection of multidirectional bending and sliding events is
demonstrated in virtual space using the self-generated triboelectric
signalsforvariousdegreesoffreedomon human hand.

GettingGroundTruths? Summary
Obviouslyyour“ground-truthgloves”cannotbeverybulkyifyouwishtoacquire
RGB(-D)simultaneouslywithnon-visionsignal
The“MITglove”quitethinalreadywith lowcost
(~$10)see3slidesabove
Itisapressure-sensorarray (550sensors)glove
though bettersuitedfor measuring“interaction
forces”,andthe force generatedbypatients,and
nota“jointtracker”
http://news.mit.edu/2019/sensor-glove-human-gr
asp-robotics-0529
Nature volume 569, pages698–702(2019)
https://doi.org/10.1038/s41586-019-1234-z
For VR applications, wearing extra
gloves / sleeves / gadgets might be
cumbersome, but for stroke/arthritis
rehabilitation, these can be used for
treatmentor asassistiveorthoses.

Glovesuitede.g.forthesetypeofmodels/applications
Generalized FeedbackLoopforJoint Hand-
ObjectPoseEstimation
MarkusOberweger ; Paul Wohlhart; VincentLepetit
IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)
https://doi.org/10.1109/TPAMI.2019.2907951
https://arxiv.org/abs/1903.10883
+https://github.com/xinghaochen/awesome-hand-pose-estimation
We propose an approach to estimating the 3D pose of
a hand, possibly handling an object, given a depth
image. We show that we can correct the mistakes made by
a Convolutional Neural Network trained to predict an
estimate of the 3D pose by using a feedback loop.
The components of this feedback loop are also Deep
Networks, optimized using training data. This approach can
be generalized to a hand interacting with an object.
Therefore, we jointly estimate the 3D pose of the hand and
the 3D pose of the object. Our approach performs en-par
with state-of-the-art methods for 3D hand pose
estimation, and outperforms state-of-the-art methods for
joint hand-object pose estimation when using depth images
only. Also, our approach is efficient as our implementation
runsin real-timeonasingle GPU.
This work can be extended in several ways. Given the
recent trend in 3D hand pose estimation, it would be
interesting to adapt the feedback loop to color
images, which means that the approach also needs to
consider lighting and texture. Further, considering a
generalization to an object class or different hand
shapes would be interesting and could be achieved by
adding a shape parameter to the synthesizer CNN. It would
also be interesting to see how this approach works with
a 3D hand CAD model instead of the synthesizer CNN.
Future work could also consider the objective criterion
of the updater training such that it would not require the
hyperparametersfor addingposes.

Handdatasetsweremissingwithobjectinteractions
ContactPose: A Dataset of Grasps with
Object Contact and Hand Pose
SamarthBrahmbhattetal.(19July2020)
https://contactpose.cc.gatech.edu/
Grasping is natural for humans. However, it involves
complex hand configurations and soft tissue deformation
that can result in complicated regions of contact
between the hand and the object. Understanding and
modeling this contact can potentially improve hand models,
AR/VR experiences, and robotic grasping. Yet, we currently
lack datasets of hand-object contact paired with
other data modalities, which is crucial for developing and
evaluatingcontactmodelingtechniques.
We introduce ContactPose, the first dataset of hand-
object contact paired with hand pose, object pose, and
RGB-D images. ContactPose has 2306 unique grasps of
25 household objects grasped with 2 functional intents by
50 participants, and more than 2.9 M RGB-D grasp images.
Analysis of ContactPose data reveals interesting
relationshipsbetweenhandposeandcontact.
Motion is recorded by 3 Kinect v2 RGB-D cameras (used
for hand pose) and an Optitrack motion capture (mocap)
system(used for objectpose).Next, they hand theobject to
a researcher, who places it on a turntable, handling it with
gloved hands. The object is recorded with the mocap
system, Kinect v2, and a FLIR Boson 640 thermal
cameraastheturntablerotatesacircle.

Ground
Truths
Howisfull-body
motioncapture
datasetsarecaptured
Moreworkfor these,
sotakeideas
forhandtracking

Ideafordatasetcreationpipelinerequirements
Wearables,BiomechanicalFeedback,and
HumanMotor-Skills’Learning&Optimization
XiangZhang,GongbingShan,YeWang,BingjunWan
andHuaLi
Appl.Sci.2019,9(2),226;
https://doi.org/10.3390/app9020226
It is well known that, among all human physical activities,
sports and arts skills exhibit the most diversity of motor
control. The datasets that are available for developing
deep learning models have to reflect the diversity,
because the depth and specialization must come from
training the deep learning algorithms with the massive
and diverse data collected from sports and arts motor
skills.
Therefore,at present,the vitalstep for developingreal-
time biomechanical feedback tool is to
simultaneously collect alargeamountofmotiondata
using both 3D motion capture (e.g., the two-chain
model with ~40 markers) and wearable IMUs (e.g., the
samemodelwithsixIMUs).
The datasets should cover large variety of sports
skills and arts performances. As such, the 3D
motion-capture data can be served as a
“supervisor” for training network model to map
IMUs data to joints’ kinematic data. Such a deep learning
model could be universally applied in motor learning and
thetraining ofsportsandartsskills.
Machineand deeplearningforsport-specificmovementrecognition:asystematicreview
of modeldevelopmentandperformance AnargyrosWilliamMcNally,Alexander Wong,John
McPheehttps://doi.org/10.1080/02640414.2018.1521769

AffordableOpticalMotionCapture vs.Vicon“GroundTruth”
Affordableclinicalgaitanalysis:Anassessmentof
themarkertrackingaccuracyof anewlow-cost
optical3Dmotionanalysissystem
BruceCarse,BarryMeadows,RoyBowers,PhilipRowe(2013)
https://doi.org/10.1016/j.physio.2013.03.001
Citedby88 -Relatedarticles
Arigidcluster offour reflectivemarkerswasusedtocomparea
low-cost Optitrack 3D motion analysis system against two
more expensive systems (Vicon 612 and Vicon MX).
Accuracy was measured by comparing the mean vector
magnitudes (between each combination of markers) for each
system.
There are a number of shortcomings of optical 3D
motion analysis systems; cost of equipment, time required
and expertise to interpret results. While it does not address all
of these problems, the Optitrack system provides a low-cost
solution that can accurately track marker trajectories to a level
comparable with an older and widely used higher cost system
(Vicon 612). While it cannot be considered to be a complete
clinical motion analysis solution, it does represent a positive
step towards making 3DGA more accessible to wider
researchandclinicalaudiences.
Next-GenerationLow-CostMotionCaptureSystemsCanProvideComparableSpatial
AccuracytoHigh-EndSystems
DominicThewlis,ChrisBishop,NathanDaniell,GuntherPaule(2013)
https://doi.org/10.1123/jab.29.1.112
We assessed static linear accuracy, dynamic linear accuracy and compared gait kinematics from a
Vicon MX-f20 system to a Natural Point OptiTrack system. In all experiments data were
sampled simultaneously. We identified both systems perform excellently in linear accuracy tests with
absolute errors not exceeding 1%. In gait data there was again strong agreement between the two
systems in sagittal and coronal plane kinematics. Transverse plane kinematics differed by up to 3° at
the knee and hip, which we attributed to the impact of soft tissue artifact accelerations on the data.
We suggest that low-cost systems are comparably accurate to their high-end
competitors and offer a platform with accuracy acceptable in research for laboratories with a
limitedbudget.
Further work is required to explore the absolute angular
accuracy of the systems and their susceptibility to high
accelerations associated with soft tissue artifact; however, it is
likely that differences of this magnitude might be evident between
competing high-end solutions. We must also begin to explore
analog integration or synchronization with low-cost
systems, as inaccuracies here could impact significantly when
calculating jointmomentsand powersusing inversedynamics

IMUsvs. Goniometer groundtruth
Predictivetrajectoryestimationduringrehabilitativetasksin
augmentedrealityusinginertialsensors
ChristopherL.Hunt;AvinashSharma;LukeE.Osborn;RahulR.Kaliki;
NitishV.Thakor DepartmentofBiomedicalEngineering, Johns Hopkins University / Infinite Biomedical Technologies
2018 IEEE Biomedical Circuits and SystemsConference (BioCAS)
https://doi.org/10.1109/BIOCAS.2018.8584805
This paper presents a wireless kinematic tracking framework used
for biomechanical analysis during rehabilitative tasks in augmented and
virtual reality. The framework uses low-cost inertial measurement units
and exploits the rigid connections of the human skeletal system to provide
egocentric position estimates of joints to centimeter accuracy. On-board
sensor fusion combines information from three-axis accelerometers,
gyroscopes,andmagnetometerstoproviderobustestimatesinreal-time.
Sensor precision and accuracy were validated using the root mean square
error of estimated joint angles against ground truth goniometer high-
precision stepper motor with a 0.9◦step size (NEMA, Rosslyn, VA)
measurements. The sensor
network produced a mean estimate accuracy of 2.81° with 1.06°
precision,resultinginamaximumhandtrackingerrorof 7.06cm.
As an application, the network is used to collect kinematic information from
an unconstrained object manipulation task in augmented reality, from
which dynamic movement primitives are extracted to characterize natural
task completion in N = 3 able-bodied human subjects. These primitives are
then leveraged for trajectory estimation in both a generalized and a subject-
specific scheme resulting in 0.187 cm and 0.161 cm regression
accuracy, respectively. Our proposed kinematic tracking network is
wireless,accurate,and especiallyusefulfor predicting voluntaryactuation in
virtualandaugmentedrealityapplications.
An overview of a rehabilitation session. (A) The individual uses an augmented
reality headset to receive kinematic tasks to complete. Tasks consist of
transporting an object to and from different quadrants while possibly changing
its orientation. Sensorized tracking nodes {nRF51822 microcontroller (Nordic Semiconductor via
RedBearLab) with MPU9250 9-axis IMU with Mahony complementary filter [protocol Nordic Enhanced ShockBurst]}
are
rigidly affixed to the anatomical landmarks and are used to record multijoint
trajectories for primitive construction. (B) Once computed, these primitives are
used to predict natural, user-specific hand trajectories in subsequent
tasks. These predicted trajectories can then be rendered by the headset to
serveas anoptimalreferencefortheuser.

GoldStandardBenchmarking IMU vs. OpticalCapture
Asensor-to-segmentcalibrationmethodformotion
capturesystembasedonlowcostMIMU
NamcholChoe,HongyuZhao,SenQiu,YonggukSo
MeasurementVolume131,January2019,Pages490-500
https://doi.org/10.1016/j.measurement.2018.07.078
A sensor-to-segment calibration method for motion
capture system is proposed. Calibration principle,
procedure and program are listed. Positions of the
magnetometer correction are determined. Influence of the
magnetic and inertial measurement units (MIMU) mounting
position is evaluated. Effectiveness of the proposed method is
validatedbyopticaldevice (NDIPolarisSpectraSystem).
Coordinate
systemsin
body and vectors
of body
segments. (a)
Body local
coordinate
system (BLCS)
and body
segment
coordinate
system (BSCS),
(b) Vectorsof
bodysegments.
Asensorfusionapproachforinertialsensorsbased3Dkinematicsand
pathologicalgaitassessments:towardanadaptivecontrolof stimulationin
post-strokesubjects
B.Sijobert;F.Feuvrier;J.Froger;D.Guiraud;C.AzevedoCoste
https://doi.org/10.1109/EMBC.2018.8512985(2018)
Pathological gait assessment and assistive control based on functional electrical
stimulation (FES) in post-stroke individuals, brings out a common need to robustly quantify
kinematics facing multiple constraints. This study proposes a novel approach using inertial
sensors to compute dorsiflexion angles and spatio-temporal parameters, in order to be later used
as inputs for online close-loop control of FES. 26 post-stroke subjects were asked to walk on a
pressure mat equipped with inertial measurement units (IMU) and passive reflective
markers. A total of 930 strides were individually analyzed and results between IMU-based
algorithms and reference systems compared. Mean absolute (MA) errors of dorsiflexion
angles were found to be less than 4°, while stride lengths were robustly segmented and
estimated with a MA error less than 10 cm. These results open new doors to rehabilitation using
adaptiveFESclosed-loopcontrolstrategies in “footdrop”syndromecorrection.

Soft-tissue Artifact(STA) human body toosoftasmetrological platform
if you start throwing IMUs to the body
Quantificationofsofttissueartifactinlowerlimb
humanmotionanalysis:Asystematicreview
AlanaPeters,Brook Galna,MorganSangeux,MegMorris,
RichardBakerGait& PostureVolume 31, Issue 1, January2010, Pages1-8
https://doi.org/10.1016/j.gaitpost.2009.09.004
Conflict of interest A/Prof Richard Baker and Dr Morgan Sangeux receive
research fundingfrom Vicon (Oxford, UK).
ASimpleAlgorithmforAssimilatingMarker-BasedMotionCaptureData
DuringPeriodicHumanMovementIntoModelsofMulti-Rigid-Body
SystemsYasuyukiSuzuki,TakuyaInoue,andTaishinNomura
FrontBioengBiotechnol.2018;6: 141.Publishedonline2018Oct18.
doi: 10.3389/fbioe.2018.00141
Here we propose a simple algorithm for assimilating motion capture data during
periodic human movements, such as bipedal walking, into models of multi-rigid-
body systems in a way that the assimilated motions are not affected by STA. The
proposed algorithm assumes that STA time-profiles during periodic movements are
also periodic. We then express unknown STA profiles using Fourier series,
and show that the Fourier coefficients can be determined optimally based solely on
the periodicity assumption for the STA and kinematic constraints requiring that
any two adjacent rigid-links are connected by a rotary joint, leading to the
STA-freeassimilatedmotionthatisconsistentwiththemulti-rigid-link model.
Rigid seven-link model of human walking. (A) Positions of landmarks and rigid
seven-link model of human body. Rigid seven-link model consists of Head-Arm-Trunk
link (HAT), left and right Thigh links (l/r-T), left and right Shank links (l/r-S), and left and right
Foot links (l/r-F). Blue circles represent landmarks of each link, and each landmark
correspondstoanatomicallandmarkofhumanbody

Jointkinematicsestimationusingamulti-bodykinematicsoptimisation
andanextendedKalmanfilter,andembeddingasofttissueartefact
modelVincentBonnetetal.-Citedby7 -Relatedarticles
JournalofBiomechanicsVolume62,6September 2017,Pages148-1558
https://doi.org/10.1016/j.jbiomech.2017.04.033
To reduce the impact of the soft tissue artefact (STA) on the estimate of skeletal
movement using stereophotogrammetric and skin-marker data, multi-body
kinematics optimisation(MKO) and extendedKalmanfilters (EKF) have
been proposed. Embedding the STA model in MKO and EKF reduced the
average RMSof markertracking from 12.6to1.6mm andfrom 4.3to1.9mm,
respectively,showingthataSTAmodeltrial-specificcalibrationisfeasible.
You could look now all the
literature on spatio-temporal
tracking (pedestrians, sports,
autonomous driving, GPS trajectory,
etc.) to constrain the possible
movementofIMU units
https://scholar.google.co.uk/scholar
?as_ylo=2015&q=spatio+temporal
+tracking+deep+learning&hl=en&a
s_sdt=0,5&authuser=1
Quantificationofthree-dimensionalsofttissueartifactsinthecaninehindlimb
duringpassivestiflemotion https://doi.org/10.1186/s12917-018-1714-7
Softtissueartifactcompensation
inkneekinematicsbymulti-body
optimization:Performanceof
subject-specifickneejoint
models(2015)
https://doi.org/10.1016/j.jbiomech
.2015.09.040
Soft-tissue Artifact(STA) human body toosoftasmetrological platform
if you start throwing IMUs to the body

CaseStudy:
Arthritis
QuickIntro
Other big
use casewould
bestroke rehabilitation

EtiologyofRheumatoidArthritis
Goingwiththeflow:harnessingthepowerofthe
vasculaturefortargetedtherapy inrheumatoid
arthritis
MathieuFerrari, ShimobiC. Onuoha, Costantino Pitzalis
DrugDiscoveryTodayVolume 21, Issue 1, January2016, Pages172-179
https://doi.org/10.1016/j.drudis.2015.10.014
Rheumatoid Arthritis (RA) is a chronic, systemic, autoimmune disease
that is considered one of the most common and severe forms of
inflammatory arthritis, associated with significant morbidity and
mortality (McInnesand Schett 2011, Firestein 2003). Although its
aetiology is poorly understood, the pathogenesis and the role of
the immune system in disease progression are well defined.
Clinically, RA is characterised by swelling of the small
diarthrodial joints, stiffness, and pain that can lead to
profound disability in the long run. At a cellular level, the hypertrophic
and hyperplastic synovial membrane, assisted by pro-inflammatory
cytokine expression and proteolytic enzymes, forms an aggressive
pannus lesion at the cartilage–bone interface, mainly comprising
macrophages, fibroblasts, and osteoclasts, which infiltrate the adjoining
articularcartilage, promotingjointdestruction.
Systemic involvements of RA include cardiovascular alterations
(e.g., pericardial inflammation and vasculitis), pulmonary, psychological,
and skeletal disorders, and is generally associated with increased
disability and shortened life expectancy (
McInnesandSchett 2011, Gullickand Scott 2011).
Detectinginflammationin rheumatoidarthritis
usingFouriertransformanalysisofdorsal
opticaltransmissionimagesfrom apilotstudy
Daniel Lighter, Andrew Filer, Hamid Dehghani, Daniel Lighter, Andrew Filer, Hamid Dehghani
The Universiotyof Birmingham (United Kingdom)
J. ofBiomedical Optics, 24(6), 066008 (2019). https://doi.org/10.1117/1.JBO.24.6.066008
Rheumatoid arthritis (RA) is a common autoimmune condition
characterized by persistent inflammation in the peripheral joints leading
to damage andlong-term disability.
Around one-third ofthose diagnosed stop work onmedical
grounds within 5 yearsof initial symptom onset Barrett etal. 2000
. In 2009, the
economic impact of RA in the UnitedKingdom wasestimated to be
£560 millionperyear in direct healthcare costsanda total
additional cost to the economy from sick leave and work-related
disabilityofsome £1.8 billion ayear NAO2009 Report
.
Despite incomplete understanding of aetiology, management of
disease progression has advanced significantly in recent decades, with
a wide range of disease modifying anti-rheumatic drugs (DMARDs)
now available to RApatients, eitherassynthetic orbiologicagents.
A window of therapeutic opportunity in the first 3 months of
symptoms is now widely acknowledged van derLinden etal.(2010)
; Nellet al.2004
, during which aggressive therapy using a combination of DMARDs
improveslong-term patientoutcomes.

HealthEconomicsofArthritis
MedicalExpendituresandEarningsLosses
AmongUSAdultsWithArthritisin 2013
Louise B. Murphy Miriam G. Cisternas David J. Pasta Charles G. Helmick Edward H. Yelin
https://doi.org/10.1002/acr.23425 -Citedby53
We estimated the economic impact of arthritisusing 2013 US Medical
Expenditure Panel Survey (MEPS) data.Total national arthritis‐
attributable medical care expenditures and earnings losses
amongadultswitharthritis were $303.5 billionin 2013. High
arthritis attributable medical expendituresmight be reduced bygreater‐
effortsto reduce pain and improve function. The highearningslosses
were largely attributable to the substantially lowerprevalence of
working among those witharthritiscompared to those without,
signalingthe needforinterventionsthat keep people witharthritisin the
workforce.
Reviewofhealtheconomicsmodellingin
rheumatoidarthritis
Paul EmeryPharmacoEconomicsvolume 22, pages55–69 (2004)
https://doi.org/10.2165/00019053-200422001-00006
As the cost of drug treatment for rheumatoid arthritis (RA) constitutes
only a smallproportion of total costs of thediseaseto individuals and
society, therapeutic interventions have the potential for significant
economic benefit. To take advantage of this potential, clinicians need
to gain a global, long-term perspective on patient care. Economic
evaluations of RA therapies are critically important in influencing
decisions regarding the role of costly, but highly effective new therapies,
particularly in settings where there are financial constraints on
healthcare provisions. Such evaluations, therefore, need to be
methodologically similar with valid results to enhance their value to
cliniciansandpolicy decision-makers.
Important issues that need to be considered in developing economic
models in RA include consideration of the connection between the
prevention of radiographic progression and downstream economic
consequences, and the need to employ lifetime models wherever
possible because a long time period is necessary to determine the true
cost-effectiveness of agents that modify radiographic progression of
RA, such as etanercept, infliximab, and adalimumab. In doing so, it is
hoped that such studies will provide optimal information to facilitate
importantdecisionsonresourceallocation.

Whatto
measure from
hand behavior

HandKinematics range of motion, handgripstrength, etc.
Handrangeof motionevaluationfor
RheumatoidArthritispatientsLucianoCejnogJr.,
RobertoMarcondes, Teofilode Campos, ValeriaElui(2019)
This framework estimates angle measurements from joints computed by a hand pose
estimation algorithm using a depth sensor (Realsense SR300). Given depth maps as input,
our framework uses Pose-REN, which is a state-of-art hand pose estimation method that
estimates 3D hand jointpositions using a deep convolutional neural network.
Predictorsofhandfunctioninpatientswith
rheumatoidarthritisB Dellhag, CSBurckhardt (1995)
http://doi.org/10.1002/art.1790080106
Measures of deficit are the most useful in predicting actual hand function, whereas
measures of strengthand flexibilityare most useful forestimated hand function.
Predictorsofhandfunctioninpatientswith
rheumatoidarthritisMaryC. Hume etal.
The Journal of Hand SurgeryVolume15, Issue 2, March 1990,
https://doi.org/10.1016/0363-5023(90)90102-W
Electrogoniometric and standard methods were used to measure bothactive and functional
ranges of motion of the metacarpalphalangeal and interphalangeal joints during 11 activities
ofdailyliving.
Handkinematics:Applicationinclinical
practiceSantosh Rath (2011)
https://dx.doi.org/10.4103%2F0970-0358.85338
The pathogenesis of deformities is influenced by bio-mechanical
principles of joints and muscle function. The crippling impact of
secondary changes due to edema, soft tissue contractures, muscle
shortening and functional adaptations also have a mechanical basis.
For clinicians and hand therapists, it is necessary to understand
these fundamental principles of biomechanics to plan treatment
modalities.
Exerciseforrheumatoidarthritisofthehand
Mark AWilliams, CynthiaSrikesavan, Peter JHeine, JulieBruce,
Lucie Brosseau, NicoletteHoxey-Thomas, Sarah E Lambi(2019)
https://doi.org/10.1002/14651858.cd003832.pub3
Itisuncertain whether exerciseimproveshandfunction or pain inthe shortterm. It
probablyslightlyimprovesfunction but has littleorno difference on paininthe
mediumandlongterm. Itis uncertainwhether exercise improves grip and pinch strength
intheshort term, and probablyhas little or no difference in themediumandlong term. The
ACR50 response is unknown. People who received exercisewithadherencestrategies
wereprobablymore adherentin the mediumtermthan who did notreceive exercise, but
withlittleor no difference in the longterm. Hand exercise probablydoesnot leadto
adverseevents. Future researchshould consider handandwrist functionastheir
primaryoutcome, describe exercise following theTIDieR guidelines, and evaluate
behavioural strategies.
Afive-yearfollowupofhand
functionandactivitiesofdailyliving
inrheumatoidarthritispatients
B Dellhag1, ABjelle (1999)
https://doi.org/10.1002/14651858.cd003832.pub3
Hand function deteriorated during a 5-year period in female RA
patients. Hand disability (GAT) improved in the male RA group,
although hand impairment (grip strength, KFT) was unchanged. Over
one-fourth of each gender group had developed a new
handicap (dependence).
InstrumentsMeasuring Pain, Physical Function, or
Patient'sGlobal AssessmentinHandOsteoarthritis:
A SystematicLiterature Search
AWillemienVisser et al. (2015)
https://doi.org/10.3899/jrheum.141228
The AUSCAN, FIHOA, VAS pain, grip and pinch strength, and pain on
palpation were most frequently used and provided most supporting
evidence for good metric properties. More research has to be
performed to comparethedifferentinstruments witheach other.
https://www.pinterest.com/pin/334533078543567020/
http://www.washingtonarthritisrheumors.com/13-hand-tips-
to-increase-joint-range-of-motion/

Howto model
the handfor arthritis
management
Graphs?

AwayfromCNNs/RNNs tograph-basedActionRecognition
[26] Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. NonLocal Graph Convolutional Networks for
Skeleton-Based Action Recognition. arXiv:1805.07694 [cs], May 2018 https://arxiv.org/abs/1805.07694
[29] Yansong Tang, Yi Tian, Jiwen Lu, Peiyang Li, and Jie Zhou. Deep Progressive Reinforcement
Learning for Skeleton-Based Action Recognition. In The IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2018. https://doi.org/10.1109/CVPR.2018.00558
[34] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial Temporal Graph Convolutional Networks for
Skeleton-Based Action Recognition. In AAAI, 2018. https://arxiv.org/abs/1801.07455
Skeleton-BasedActionRecognitionwithDirected GraphNeuralNetworks
LeiShi,YifanZhang,JianCheng,HanqingLu;TheIEEE ConferenceonComputer VisionandPatternRecognition(CVPR), June2019,pp.7912-7921
http://openaccess.thecvf.com/content_CVPR_2019/papers/Shi_Skeleton-Based_Action_Recognition_With_Directed_Graph_Neural_Networks_CVPR_2019_paper.pdf
In this work, we represent both joint and bone information as a directed
acyclic graph and design a customized novel directed graph neural
network (DGNN) to predict action based on the constructed graph. In addition,
we make the graph structure adaptive to better fit the multilayer architecture and
the recognition task. Furthermore, the motion information between consecutive
frames is extracted to model the temporal information of a skeleton
sequence, and both the spatial and motion information are fused in a
two-stream framework. The final model exceeds current stateof-the-art
performance on two large-scale datasets: NTURGBD and Skeleton-Kinetics.
Future work might focus on how to exploit the skeleton data and RGB data
together. In addition, exploration is recommended into how to combine the
problem of pose estimation with skeleton-based action recognition in a unified
architecture.

SkeletonExample DefinitionsandUses
Space-timerepresentationof peoplebasedon3Dskeletaldata:Areview https://doi.org/10.1016/j.cviu.2017.01.011

DeepLearning meets DynamicNonlinearSystems: GenerativeMarkovState
DeepGenerativeMarkovStateModels
HaoWu,AndreasMardt,LucaPasquali,FrankNoe
(Submittedon19May2018(v1),lastrevised11Jan2019
(thisversion,v2))https://arxiv.org/abs/1805.07601
https://github.com/markovmodel/deep_gen_msm PyTorch/TF
WeproposeadeepgenerativeMarkovStateModel
(DeepGenMSM)learningframeworkfor inferenceof
metastabledynamicalsystemsand predictionof
trajectories.After unsupervisedtrainingontime
seriesdata,themodelcontains(i) aprobabilistic
encoderthatmapsfromhigh-dimensional
configurationspacetoasmall-sizedvector indicating
themembershiptometastable(long-lived)states,(ii)a
Markovchainthatgovernsthetransitionsbetween
metastablestatesandfacilitatesanalysisofthelong-
timedynamics,and(iii)agenerativepart that
samplestheconditionaldistributionofconfigurationsin
thenexttimestep.Themodelcanbeoperatedina
recursivefashiontogeneratetrajectoriesto
predictthesystemevolution fromadefined
startingstateand proposenewconfigurations.
TheDeepGenMSMisdemonstratedtoprovide
accurateestimatesofthelong-timekineticsand
generatevaliddistributionsfor molecular dynamics
(MD)benchmarksystems.Remarkably,weshowthat
DeepGenMSMsareabletomakelongtime-steps
inmolecular configurationspaceandgenerate
physicallyrealisticstructuresinregionsthat
werenotseenintrainingdata.
In conclusion, deep MSMs provide high-quality modelsof
the stationary and kinetic properties for stochastic
dynamical systems such as MD simulations. In contrast
to other high-quality models such as VAMPnets, the
resulting model is truly probabilistic and can thus
be physically interpreted and be used in a
Bayesian framework. For the first time, it was shown
that generating dynamical trajectories in a 30-
dimensional molecular configuration space results in
sampling of physically realistic molecular structures.
While Alanine dipeptide is a small system compared to
proteins and other macromolecules that are of biological
interest, our results demonstrate that efficient sampling of
new molecular structures is possible with generative
dynamic models, and improved methods can be built
upon this. Future methods will especially need to
address the difficulties of generating valid
configurations in low-probability regimes, and it is
likely that the energy distance used here for
generator training needs to be revisited to achieve
thisgoal.

ExampleGCNN for hand pose estimation
HOPE-Net:AGraph-BasedModelforHand-ObjectPose
Estimation
BardiaDoosti,ShujonNaha,MajidMirbagheri,DavidJ.Crandall;
ProceedingsoftheIEEE/CVF Conferenceon ComputerVision andPatternRecognition
(CVPR),2020,pp.6608-6617
https://openaccess.thecvf.com/content_CVPR_2020/html/Doosti_HOPE-Net_A_Grap
h-Based_Model_for_Hand-Object_Pose_Estimation_CVPR_2020_paper.html
https://github.com/bardiadoosti/HOPE Torch
Hand-object pose estimation (HOPE) aims to jointly
detect the poses of both a hand and of a held
object. In this paper, we propose a lightweight model
called HOPE-Net which jointly estimates hand and
object pose in 2D and 3D in real-time. Our network uses a
cascade of two adaptive graph convolutional
neural networks (GCNN), one to estimate 2D
coordinates of the hand joints and object corners, followed
by another to convert 2D coordinates to 3D. Our
experiments show that through end-to-end training of the
full network, we achieve better accuracy for both the 2D
and 3D coordinate estimation problems. The proposed 2D
to 3D graph convolution-based model could be applied to
other 3D landmark detection problems, where it is possible
to first predict the 2D keypoints and then transform them to
3D.

Addfeatures
beyondjointcoordinates?

2s-AGCN bonelengthsasregularizers
Two-StreamAdaptiveGraphConvolutionalNetworksforSkeleton-Based
ActionRecognitionLeiShietal.CVPR2019
National Laboratoryof PatternRecognition,Instituteof Automation,ChineseAcademyof Sciences;University of Chinese
Academyof Sciences; CAS Centerfor ExcellenceinBrainScienceandIntelligenceTechnology
https://github.com/lshiwjx/2s-AGCNPyTorch
In skeleton-based action recognition, graph convolutional networks
(GCNs), which model the human body skeletons as spatiotemporal
graphs, have achieved remarkable performance. However, in
existing GCN-based methods, the topology of the graph is set
manually, and it is fixed over all layers and input samples. This may not
be optimal for the hierarchical GCN and diverse samples in action
recognition tasks. In addition, the second-order information (the
lengths and directions of bones) of the skeleton data, which is
naturally more informative and discriminative for action recognition, is
rarely investigated in existing methods. In this work, we propose
a novel two-stream adaptive graph convolutional network
(2s-AGCN) for skeleton-based action recognition. The topology of the
graph in our model can be either uniformly or individually learned
by the BP algorithm in an end-to-end manner. This data-driven method
increases the flexibility of the model for graph construction and brings
more generality to adapt to various data samples. Moreover, a two-
stream framework is proposed to model both the first-order and the
second-order information simultaneously, which shows notable
improvementforthe recognitionaccuracy.

SwellinginRA node encoding?
MuscleActivation/stiffnessinRA edgeencoding?
ScalableMuscle-actuatedHuman
Simulationand Control
http://mrl.snu.ac.kr/research/ProjectScalable/Page.htm
->https://github.com/lsw9021/MASS
https://www.youtube.com/watch?v=kie4wjB1MCw
F8 2019:ResearchTowardsVRFullBodyTracking& Avatars
https://youtu.be/FhiAFo9U_sM -
https://uploadvr.com/facebook-f8-2019-body-tracking/

How to define Swelling inRA?
Measuringdiseaseactivity andresponse totreatment in
rheumatoidarthritis
CasandraBuzatu andRobertJ. Moots
https://doi.org/10.1080/1744666X.2019.1559050
2.Jointcounts
Initially, assessments of disease activity and response to
treatment were based solely on the patient’s and doctor’s overall
perception, in a non-quantifiable way. However, because articular
involvement is the main feature of RA, various tender and swollen
joint counts were developed including the Ritchie articular
index (grades the tenderness of 53 joints), American
Rheumatism Association index (counts joints that are
tender/swollen), and the Lansbury articular index (scores the
severity of inflammation in accordance with joint surface) [
Thompsonetal.1987, ACR].
The swollen joint counts correlated closely with acute phase
reactants, while the tender joint count mapped with patients’
scores on pain scales [e.g. Thompsonetal.1991]. There was
therefore a clear need to identify better systems, able to
effectively incorporate both objective and subjective
measurements. This led to the development of composite
scoresfordisease activity onRA.
https://www.health.harvard.edu/a_to_z/rheumatoid-arthritis-a-to-z
https://www.mayoclinic.org/diseases-conditions/rheumatoid-arthriti
s/symptoms-causes/syc-20353648
Note!The
pathological
hands,that
might ormight
notcauseissues
onhandtracking
modelstrained
fornon-
pathological
hands

Measuring gripstrengthandhandfunction?
NHSNIHRSouthamptonBiomedical
ResearchCentreProcedurefor Measuring
HANDGRIPSTRENGTHUSINGTHE
JAMARDYNAMOMETER
https://www.uhs.nhs.uk/Media/Southampton-Clinical-Researc
h/Procedures/BRCProcedures/Procedure-for-measuring-grip
strength-using-the-JAMAR-dynamometer.pdf
Research findings: routinemeasurementofgripstrengthcan
helpfrailty (Ibrahimetal.2019DOI: 10.1186/s12877-018-0768-5)
https://www.csp.org.uk/frontline/article/research-findings-routine-measurement-grip-stren
gth-can-help-frailty
Grippit https://doi.org/10.1080/10749357.2016.1168591
Finger extensionforce measurement device(EX-it)
https://pubmed.ncbi.nlm.nih.gov/18776605/
CreateLabz| MeasuringHandGripStrengthwithFlexSensor
onBluetoothwithArduino.community.createlabz.com

Howto model
the handfor arthritis
management
NeuralODEs
and other
dynamical
time-series
methdos

NeuralODEs are sexy #1
AugmentedNeuralODEs
https://arxiv.org/pdf/1904.01681.pdfUniversityofOxford
LatentODEsforIrregularly-SampledTimeSeries
https://arxiv.org/pdf/1907.03907.pdf
UniversityofTorontoandtheVectorInstitute
NeuPDE:NeuralNetworkBased
OrdinaryandPartialDifferential
EquationsforModelingTime-
DependentData
https://www.technologyreview.com/2018/12/12/1739/a-ra
dical-new-neural-network-design-could-overcome-big-ch
allenges-in-ai/

NeuralODEs? #2
Model-basedReinforcementLearningforSemi-
MarkovDecisionProcesseswith Neural ODEs
JianzhunDu,JosephFutoma,FinaleDoshi-Velez Harvard University
(29Jun2020)
We present two elegant solutions for modeling continuous-
time dynamics, in a novel model-based reinforcement learning
(RL) framework for semi-Markov decision processes (SMDPs),
using neural ordinary differential equations (ODEs). Our
models accurately characterize continuous-time dynamics and
enable us to develop high-performing policies using a small
amountof data.
TheLatent-ODE(Rubanovaetal.2019,github.com/YuliaRubanova/latent_ode
)ismoresample-efficientthan
other modelsandthemodel-freemethodonallthreetasks.Forexample,ontheswimmer
task,wedevelopahigh-performing policyover100kenvironmentstepsusingtheLatent-
ODE,whereasthemodel-freebaselinerequiresfourtimestheamountofdata.However,the
ODE-RNNisnotasgoodastheLatent-ODE anditsperformanceissimilar withother
baselinemodels.
https://youtu.be/YZ-_E7A3V2w
David Duvenaud Jan 2020
“I'm afraid this talk gave
some people the
impression that I was
flippant about careful
scholarship, or wrote a
misleading paper with no
substance. I want to be
clear that I do take these
things seriously, and that
the paper made substantial
contributions which still
stand. The message I
intended was that *even
though we tried hard to be
careful*, we still made
mistakes, which we
eventually fixed.”
https://twitter.com/luckecianomelo
Mujocotasks
http://www.mujoco.org/

NeuralODEs? #3
ODE2
VAE:Deep generative second order
ODEswith Bayesianneural networks
ÇağatayYıldız,MarkusHeinonen,HarriLähdesmäki AaltoUniversity
(Submittedon27May2019)
https://arxiv.org/abs/1905.10994 Cited by 10

https://github.com/cagatayyildiz/ODE2VAE
WepresentOrdinary DifferentialEquation VariationalAuto-Encoder (ODE2
VAE), a latent secondorder ODE model for high-
dimensional sequential data. Leveraging theadvances in deep generative models, ODE2
VAE can simultaneouslylearn the embeddingof
highdimensional trajectories and inferarbitrarilycomplexcontinuous-timelatentdynamics. Our model explicitly
decomposes thelatentspaceinto momentum and position components and solves a second order ODE system, whichis in contrast to
recurrent neural network (RNN)based timeseries models and recently proposed non-parametric ODE techniques. In order to accountfor
uncertainty, wepropose probabilistic latent ODE dynamics parameterized bydeep Bayesian neural networks. Wedemonstrate our
approachon motion capture, image rotationandbouncing balls datasets. We achievestate-of-the-art performancein longterm motion
predictionand imputation tasks.
Neural Spline Flows
Conor Durkan,Artur Bekasov,IainMurray,GeorgePapamakarios
(Submittedon10Jun2019)
https://github.com/bayesiains/nsf
A normalizing flow models a complex probability density as an invertible transformation of a simple
base density. Flows based on either coupling or autoregressive transforms both offer exact density
evaluation and sampling, but rely on the parameterization of an easily invertible elementwise
transformation, whose choice determines the flexibility of these models. Building upon recent work, we
propose a fully-differentiable module based on monotonic rational-quadratic splines,
which enhances the flexibility of both coupling and autoregressive transforms while retaining analytic
invertibility. We demonstrate that neural spline flows improve density estimation, variational
inference, and generativemodelingof images.
Continuous-time flows Rather than constructing a normalizing flow as a series
of discrete steps, it is also possible to use a continuous-time flow, where the
transformation from noise u to data x is described by an ordinary differential
equation. Deep diffeomorphic flow is one such instance, where the model is
trained by backpropagation through an Euler integrator, and the Jacobian is
computed approximately using a truncated power series and the Hutchison trace
estimator. Neural ordinary differential equations [Neural ODEs] define an
additional ODE which describes the trajectory of the flow’s gradient, avoiding
the need to backpropagate through an ODE solver. A third ODE can be used to
track the evolution of the log density, and the entire system can be solved with a
suitable integrator.
The resulting continuous-time flow is known as FFJORD. Like flows based on
coupling layers, FFJORD is also invertible in ‘one pass’, but here this term
refers to solving a system of ODEs, rather than performing a single neural-
network pass.
CMU motion capture library
There are several directions in which our work can be
extended. Considering divergences different than KL
would lead to Wasserstein auto-encoder formulations.
The latent ODE flow can be replaced by stochastic flow,
which would result in an even more robust model.
Proposed second order flow can also be combined with
generative adversarial networks to produce real-looking
videos.

Hand
tracking
itself gets
youonly sofar
’Digitalhealth’
solutionsand
other measures
for
Arthritis

Arthronica(London,England) A SaaS platformtoremotely diagnose arthritisusing laptop/smartphone
camerascombined with AI-powered software and providesrapid accesstodataon illnessprogressionto
optimize patientrecoverypathways. http://www.arthronica.com/

A pilot study of 20 patients at Salford Royal Hospital examined how data captured on a
smartphone app could improve doctors’ consultations when integrated into
the electronic health record (EHR). Patients input their symptoms into the app
each day for three months and recorded the impact they had on their lives. This was
then uniquely integrated into the hospital EHR and summarised as a graph, which was
visible atoutpatientvisits.Researchersfoundthe app captured flares andlong-term
trends in symptoms that could otherwise have been missed and improved the
experienceofface-to-faceconsultationsfor patientsanddoctors.
The study published in the journal Rheumatology was jointly funded by Versus Arthritis
and the National Institute for Health Research Collaboration for Leadership in Applied
HealthResearchandCare(NIHRCLAHRC)GreaterManchester.
THELARGERCONTEXT
Meanwhile, pharmaceutical giant Pfizer has teamed up with Finnish health tech
startup Popit to providesupport to people taking rheumatoid arthritis medication.
Pfizer patients in Finland, Sweden and Norway will be offered Popit’s adherence
solution, which monitors pill-taking with a smart device and alerts users via an app if
theyforgettotaketheirmedication.
Earlier this year, digital health company Living With announced a partnership with
the Royal United Hospitals Bath NHS Foundation Trust and the University of Bath to
develop a platform called the Rheumatoid Arthritis Flare Profiler (supported by
Innovate UK grant). The project will allow patients capture disease activity
data on their smartphones and use machine learning to create an effective
treatmentplan.
https://www.healthcareitnews.com/news/europe/daily-remote-monitoring-rheum
atoid-arthritis-patients-can-improve-doctor-consultations
https://www.healthtechdigital.com/living-with-wins-innovate-uk-grant-to-develop-a-
rheumatoid-arthritis-flare-profiler-2/

https://www.elsa.science/english
Norrskenisleadinga€2mseedroundin ElsaScience,aSwedishstartupthathas
createdadigitalplatformtoprovideindividualisedhealthadvice for people
sufferingwithrheumatoidarthritis—adiseasethataffects about1%oftheworld
population.
Itsappletspeoplekeeptrack oftheir symptoms,dietandlifestyle,aswellashowthey
areresponding tomedication.Itsusersareguidedby“Elsa”—apersonaliseddigital
assistant,whogivessuggestionsbasedontheirindividualcircumstances. Thegoalis
to“democratisescientificresearch”,accordingtothestartup’scofounder andchief
executiveSofiaSvanteson,whosaidthatthereisahugeamountofresearchthatis
trickyforordinarypeopletoaccessandactupon.
https://sifted.eu/articles/elsa-science-norrsken-fund/

‘Digitalhealth’approachesgettingpopularforarthritisaswell
Pharmanaturallyinterested
DevelopingSmartphone-BasedObjective
AssessmentsofPhysical Functionin
RheumatoidArthritisPatients:ThePARADE
Study
Valentin HamyDigital Biomarkers, R&D Development, GlaxoSmithKline, Stevenage,United Kingdom
Digital Biomarkers2020;4:26–43 https://doi.org/10.1159/000506860
Digital biomarkers that measure physical activity and mobility are of
great interest in the assessment of chronic diseases such as rheumatoid
arthritis, as it provides insights on patients’ quality of life that can be reliably
compared across a whole population. To investigate the feasibility of
analyzing iPhone sensor data collected remotely by means of a mobile
software application in order to derive meaningful information on functional
abilityin rheumatoidarthritispatients.
Motion-specific features including wrist joint range of motion (ROM) in
flexion-extension (for thewrist motion test) andgait parameters(forthewalk
test) were extracted from high quality data and compared with subjective
pain and mobility parameters, separately capturedvia the application. Out of
646 wrist joint motion samples collected, 289 (45%) were high quality. Data
collected for the walk test included 2,583 samples (through 867 executions
ofthetest) fromwhich 651(25%) werehigh quality.
These findings demonstrate the potential to capture and quantify
meaningful objective clinical information remotely using iPhone
sensors and represent an early step towards the development of patient-
centricdigitalendpointsforclinicaltrialsinrheumatoidarthritis.
ObservationalStudy ofaWearableSensorand
SmartphoneApplicationSupporting
UnsupervisedExercisestoAssessPainand
Stiffness
PerraudinC.G.M Digital Development, Novartis Pharma AG, Basel,Switzerland
Digital Biomarkers2018;2:106–125 https://doi.org/10.1159/000493277
Evaluation of pain and stiffness in patients with arthritis is largely based on
participants retrospectively reporting their self-perceived pain/stiffness. This is
subjective and may not accurately reflect the true impact of therapeutic
interventions. We now have access to accelerometer-based systems
ActiGraph GT9XLink
to continuouslycapture objective information regarding movement
andactivity.
This study demonstrates the feasibility and usefulness of regular, sensor-based,
monitored,unsupervised physical teststo objectively assess the impact of
disease onfunction inthehomeenvironment.Thisapproach maypermit
remotedisease monitoringinclinicaltrials andsupportthedevelopment
ofnovelendpointsfrompassivelycollectedactigraphydata.
https://www.outsourcing-pharma.com/Article/2018/12/04/The-top-wearables-for-clinical-research

Commentaryarticles to getstarted
Rheumatology4.0:bigdata,wearablesand diagnosisby
computer
GerdRBurmesterDepartment ofRheumatology and ClinicalImmunology,Charité–University MedicineBerlin,Berlin,
Germany
AnnalsofRheumaticDiseasesJuly2018
http://dx.doi.org/10.1136/annrheumdis-2017-212888
Artificial Intelligence (AI) andrheumatology:a potential
partnership
SuruchiKothari,LetiziaGionfrida,AnilAnthonyBharath,SonyaAbraham
Rheumatology,Volume58,Issue11,November 2019,Pages1894–1895
https://doi.org/10.1093/rheumatology/kez194
Bigdataanddataprocessinginrheumatology:
bioethicalperspectives
AmarantaManriquedeLara&IngrisPeláez-Ballestas
ClinicalRheumatologyvolume39,pages1007–1014(2020)
https://doi.org/10.1007/s10067-020-04969-w
“Mobile devices will potentially extend the reach of specialists outside of the clinic setting. A forecast is that 6.3
billion smartphone subscriptions will exist by the year 2021 and can therefore potentially provide inexpensive
universal access to diagnostic care. There will also be a tremendous progress in wearable technology. Implantable
devices to store data are available as well and whole genome typing will be quite inexpensive in the future. Instant
scanning methods to assess inflammation already document involved joints, currently using fluorescence optical
imaging. Proteomicsand autoantibodyanalysiscan bedone withsophisticated and rapid techniques.
“In rheumatology, high throughput technologies in molecular research already generated big data in
rheumatology some 15 years ago. These included ‘omics’ technologies, such as genomics, transcriptomics and
cytomics. In the future, functional analysis and interpretation will require adaptation of existing and the
development of new software toolsfrequentlybased onmachine learning.”
“It is anticipated that the applications of AI and ML are set to advance in the field of rheumatology.
Factors driving this transformative technology and its increased adoptions include progress in wearable
technologies and their ability to track activity and cardiovascular data, with the possibility of extension to the
monitoring of serum parameters, including inflammatory markers such as CRP. These technologies have
also been used to monitor drug adherence with ingestible sensors. The decreasing costs, increasing speeds
and rising use of genetic sequencing will further drive the adoption of AI. AI will ultimately be a tool in the
rheumatologists’s kit, which among other things, has the potential to support research and clinical care, and
perhaps even ease some of the bureaucratic burdens. With these caveats in mind and ultimate goals in mind,
further considerationsintoapplyingAItechnologiescanbe sought.”
“Overall, we need to develop and implement new ethical and regulatory models,
responding to collective experiences in the implementation of big data and AI into
rheumatology so far. This review aims to serve as a basis for future debate on the bioethical
implications of big data in rheumatology and health in general. This kind of
ongoing discussion can help to discern whether all new applications of big data in
healthcare are ethical: just because technology enables us to do something does not
necessarilymeanweshoulddoit.”

Biomarkers
startups and→
emergingtech
to keep
an eye on
Build multimodal→
progression models
withoutthe‘magical
biomarker’ to explain
it allapproach

Basicideatracktheprogressionofdiseaseinclinicalsetting,pharmawantsto
tracktheefficacyoftheirtrials
Measuringdiseaseactivity andresponse totreatment in
rheumatoidarthritis CasandraBuzatu andRobertJ.Moots
https://doi.org/10.1080/1744666X.2019.1559050
Effective treatment of rheumatoid arthritis (RA) requires suppression
of the underlying inflammation. Measurement of such
inflammation, the disease activity, is mandatory to target treatment
andmaximize outcomes. However, thisisnot as straightforward
asitmayseem.
The key to effective management of RA is the rapid
suppression of inflammation, ideally to remission, with
maintenance of such remission. The aim is to prevent disability and
maximize quality of life. Central to this is the ability to determine
disease activity (potentially open to suppression) as opposed to
damage (irreversible).
A variety of measures are currently available, allowing better
assessmentof response to treatment. Perhaps, the elephant in the
room in treating RA is the current hit and miss choice of
therapy. Whilst inflammatory disease can be suppressed, and fully,
the optimal drug is selected through trial and errorIn the future, the
development of predictive biomarkers (e.g. Wrightetal. 2017)
allowing targeting of drugs may revolutionize this field and render the
toolsof todayredundant.
Classification ofdiseaseactivityaccordingtoDAS28-ESR.

Notjustaboutgettingsignalsfromdevices: Syndemics
Treatmentfailureininflammatory
arthritis:timetothinkaboutsyndemics?
ElenaNikiphorouKing’s College London
, HeidiLempp, Brandon A. Kohrt
Rheumatology, Volume 58, Issue 9, September 2019, Pages1526–1533,
https://doi.org/10.1093/rheumatology/kez222
Social determinants of health play a crucial role in health
and disease. In current times, it has become increasingly known
that biological and non-biological factors are potentially
linked and help to drive disease. For example, links between
various comorbidities, both physical and mental illnesses, are
known to be driven by social, environmental and economic
determinants. Thiscontributesto worse disease outcomes.
This article discusses the concept of syndemics, which
although well-described in some conditions, represents a novel
concept in the context of rheumatic and musculoskeletal
diseases. Written in the form of a viewpoint, the article focuses
on a novel theoretical framework for studying
inflammatory arthritis, based on a syndemic approach that
takes into account the social context, biocultural disease
interaction, and socio-economiccharacteristicsof the setting.
Syndemics involving inflammatory arthritis may be most likely
in a social context involving limited access to health care,
lack of physical activity and obesogenic diets, high rates
of alcohol consumption, and high exposure to stressful
life events.

OmicsBiomarkersforRheumatoidArthritis(RA)
RNAIdentificationof PRIME CellsPredictingRheumatoidArthritis
Flares DanaE.Orangeetal.(July16,2020)
NEnglJ Med2020;383:218-228
http://doi.org/10.1056/NEJMoa2004114
http://doi.org/10.1126/science.abd9156
Longitudinal genomic analysis of rheumatoid arthritis flares revealed
preinflammatory mesenchymal (PRIME cells) in the blood during the
period before a flare and suggested a model in which these cells become
activatedby Bcellsintheweeksbeforeaflareandsubsequentlymigrateoutof
the blood into the synovium. (Funded by the National Institutes of Health and
others.)
The researchers also found RNA signatures associated with immature white
blood cells that peaked in the days prior to PRIME cell activity. White blood cells are
responsible for initiating cascades of immune activity—recruiting other inflammatory
cells in order to do so—and dysfunctional white blood cells have been implicated in
multiple autoimmune diseases. Taken together, the finding suggests PRIME cells
couldbe mobilized by abnormalimmune system activity,Darnell says.
Prior research has found that similar inflammation-causing connective tissue cells
cause the disease in the jointsofmice, but this is the first time such cells have
been found in the human bloodstream. That makes this study “important,” as it
could direct researchers’ attention to ways in which these cells could be manipulated
to rein in the disease, says Robert Winchester, a rheumatologic pathologist at
Columbia University. However, the study is just a first step, Elewaut cautions.
Knowing for certain how PRIME cells influence flare-ups will require “a much larger
set” of experiments.

DNAlyticsRheumaKit
RheumaKit,a testthatanalyzesbiological and clinical data to
identifyand treatarthritis disease.
RheumaKit allows a doctor to identify exactly what kind of
arthritis a patient has. Then it goes further by suggesting the
mostappropriate treatmentsmechanisms.
Here is how it works: a biopsy is harvested from joint
tissue, and the sample is sent to our analytic lab. An RNA
analysis is done by qPCR. In addition, the doctor has to
answer a series of binary questions that add more
information. We cross-reference all these data, and 15 days
later, the doctor receives the complete report. The doctor also
gets the picture of the metabolic pathways, shedding light on
whichtreatmentmightbe the bestmatch, case bycase.
There is a dozen of such very expensive treatments, with
different mechanisms of action and all with about 60% efficacy.
As there is no way for the rheumatologist to objectively choose
between these treatments, a trial-and-error mechanism is often
initiated. The patient loses years, and the governments pay
thousands of euros for nothing. This could be avoided with our
solution.

InnVentis handwavey
http://www.innventis-pharma.com/platform.php

Imagingforrheumatoidarthritis
Examplesimagesfor clinical andexperimental methodsofimagingthe
effectsofrheumatoidarthritisin the handjoints.Figuresa-edepictx-ray
(Stanislavsky),MRI (Navalho etal. 2012),ultrasound (Rheum.), near-infrared
fluorescenceimaging (Glimmetal. 2015)and frequency-domaindiffuseoptical
tomography (Montejoetal. 2012)respectively,reproduced fromthecited
sources.https://etheses.bham.ac.uk/id/eprint/9008/
Examples of in vivo imaging for
symptoms of joint inflammation in
rodent models of rheumatoid arthritis.
Consecutively, these images depict a)
ultrasound imaging of a mouse ankle joint
(Claveletal.2007) b) radiolabelled
microSPECT/microCT targeting integrin
v 3 expression on osteoclasts in CIAαvβ3 expression on osteoclasts in CIA β3 expression on osteoclasts in CIA
mice () c) T1 microMRI of a normal rat
ankle (Jacobsonetal.1999) and d) non-
specific near-infrared fluorescence
imagingofaCIArat(Vollmeretal.2014).
ThermographicImagingGauges
Rheumatoid ArthritisActivity
https://www.medimaging.net/general-imaging/arti
cles/294780164/thermographic-imaging-gauges-
rheumatoid-arthritis-activity.html
https://www.vision-systems.com/lighting-optics/article/1403
9587/improved-rheumatoid-arthritis-detection-by-fouriertra
nsform-analysis
→ https://doi.org/10.1117/1.JBO.24.6.066008 from
Universityof Birmingham
Assessing spectral imaging of thehuman
finger fordetection of arthritis
https://doi.org/10.1364/BOE.10.006555

Radiologicalbiomarkers
Conventionalradiographyof the handsandwristsin
rheumatoidarthritis.What arheumatologistshouldknow
andhowtointerpret the radiologicalfindings
AlexandrosA.Drosos,EleftheriosPelechas&ParaskeviV. Voulgari
RheumatologyInternationalvolume39,pages1331–1341(2019)
https://doi.org/10.1007/s00296-019-04326-4
Rheumatoid arthritis(RA) isachronic inflammatorydiseaseaffectingthesynovial
membrane, leading to joint damage and bone destruction. Conventional
radiography (CR, as in X-Ray) of the hands and wrists has been, for many
years, the primary imaging modality used to diagnose and monitor RA. On the
other hand, many investigatorsin clinical trials and observationalstudies used CR
of the hands and wrists to demonstrate drug effectiveness and structural
damageprogression.
Therefore, CR is better when utilized as a serial assessment over time to
determine disease progression. CR remains the most commonly used imaging
tool in rheumatology and has a number of advantages: (a) it is easily available in
most rheumatologistsandreadily accessibleto most patients, (b) it isinexpensive
and relatively safe, (c) it provides immediate information and can be interpreted
easily by therequestedrheumatologist and, (d) thedataarereproducibleandcan
be used for serial evaluation and follow-up. On the other hand, CR does have
some limitations: (a) it uses ionizing radiation, (b) does not provide good
information in early RA patients, and, (c) does not provide information about
synovial inflammation or other soft tissue structures. However, CR remains
an important imaging technique for the evaluation of RA patients and those with
peripheralarthritis.
Soft tissueswellingisnot a
pathognomonic findingofRA[7].
Symmetrical soft tissue swellingis seen
in earlyphase of RAbut alsoin other
inflammatory arthropathies[23].

Radiologicalbiomarkers Whatcanyouseeintheimage(s)
Applicationof anadvanced noisereductionalgorithmfor
imagingofhandsinrheumaticdiseases:evaluationofimage
quality comparedtostandard dose images‑dose images
KatharinaZiegeleretal.(2020)
RheumatologyInternational(2020)40:893–899
https://doi.org/10.1007/s00296-020-04560-1
X-ray is the fundamental imaging technique in both diagnosis and
follow-up of rheumatic diseases. As patients often require sequential
X-rays over many years, dose reduction is of great importance.
New advanced noise reduction algorithms allow for a dose reduction
of up to 50%. The aim of this study was to evaluate whether quality of
low-dose images is non-inferior to standard-dose images and,
therefore, application of this technique is possible in the context of
imagingof rheumaticdiseases.
… limitations … Information on disease duration and severity was
not available due to the very heterogeneous patient population in all
groups, so that some variation may be possible. Lastly, any
semiquantitative assessment such as applied in this investigation is
prone to some degree of bias from subjectivity, which cannot be
entirely excluded. Overall, our results show that application of the noise
estimation based low-dose technology in the context of imaging of
rheumatic diseases is feasible, allowing for a reduction of radiation
exposure by up to 50%. Further studies on quantitative, rather
than qualitative,measuresofimage qualityarewarranted.

Radiologicalbiomarkers addultrasoundforimagingandtreatment#1
MRandultrasoundofthe handsand wrists inrheumatoid
arthritis.PartII.Added clinicalvalue
DavidA.RubinSkeletal Radiology volume48,pages837–857(2019)
https://doi.org/10.1007/s00256-019-03180-6
Part 2 will now explore how these imaging features impact patient
management. Specific topics will include how MR and US imaging
findings not a single US image actually on review?
can facilitate an earlier and more
specific diagnosis than is possible with just clinical and serologic
markers, can help prognosticate the aggressiveness of the disease
course, can assess disease activity at both symptom onset and during
medical treatment, and can contribute to identifying remission. Lastly, this
review will discuss potential future roles of imaging in preclinical
disease (when risk factors are present without symptoms), and for
directing personalized medicine (selecting drugs that are more likely
to be effective in specific patients).
A bigger question is whether incorporating MR or US for the diagnosis
of early RA has actualadded value or is cost-effective. Here, the juryis
still out. A cohort study of early arthritis patients found that incorporating
US findings into a clinically based prediction tool did not improve the
ability to predict an outcome diagnosis of RA [Pratt et al.2013]. On the
other hand, adding MR criteria (symmetric synovitis, osteitis, or
erosions) to the clinically based 2010 American College of Rheumatology/
EULAR classification criteria (Table) does increase sensitivity for
predicting conversion from UA to RA (defined as meeting the 1987
classification criteria or the initiation of DMARD therapy based on clinical
judgment) by 1 year, but at a cost of decreased specificity and slightly
decreased PPV
CanUltrasoundTherapy Help MyRheumatoidArthritis?
https://www.healthline.com/health/rheumatoid-arthritis/ultrasound-therapy
MedicallyreviewedbyNancyCarteron,MD,FACR —WrittenbyColleenM.Story—UpdatedonJanuary9,2017
Therapists sometimes use ultrasound therapy to help reduce inflammation
and pain. In 2002, Casimiro et al.2002 Citedby148
published a study in the
Cochrane Database of Systematic Reviews on ultrasound therapy in people
with RA. The study suggests that when ultrasound is applied to your
hands, itmayhelpincrease your grip strength. It mayalso help:
●
improve wrist flexibility
●
decrease morningstiffness
●
reduce the number of swollen and painful joints
Despite these results, more research is needed on the use of ultrasound
therapyforRA. High-qualityclinical trialson the subject are lacking.
In 2009, Erdogan and Esen published a study in the Journal of Ultrasound in
Medicine on ultrasound therapy and bone healing. The researchers
reviewed older and new literature findings. Some studies showed links
between ultrasoundand bone healing.
The authors didn’t focus specifically on RA. But the bone-healing potential of
ultrasound therapy might help people who experience bone erosion or other
deformitiesasa complication of RA.

Radiologicalbiomarkers addultrasoundforimagingandtreatment#2
Swollen,butnottenderjoints,are independentlyassociated
withultrasoundsynovitis:resultsfromalongitudinal
observational studyofpatientswithestablishedrheumatoid
arthritis
Hammeretal.(2019)Department of Rheumatology,Diakonhjemmet Hospital,Oslo,Norway
http://dx.doi.org/10.1136/annrheumdis-2019-215321
Joint swelling and tenderness are considered a proxy for inflammation in patients
with rheumatoid arthritis (RA). With ultrasound-detected inflammation as reference, our
objectives were to explore on patient and joint level the associations between ultrasound
synovitisandjointswelling,tendernessandpatient-reportedjointpain(PRJP).
Swollen joints were strongly associated with ultrasound detected synovitis at both patient
and joint level, while this association was not found for tender joints. These results may
questioniftenderjointsreflectongoinginflammation inestablishedRA.
Whatdoesthisstudyadd?
●
This is the largest longitudinal study exploring at both patient and joint level the
associations between synovitis assessed by ultrasound (as reference) and tender joints,
swollen jointsaswellas patient-reportedjointpain.
●
Tender joints and patient-reported joint pain were primarily associated with patient-reported
outcomes, whileswollen jointswereprimarily associatedwithultrasound synovitis.
Howmight thisimpact on clinicalpracticeorfuturedevelopments?
●
The present study of patients with established RA finds tender joints to have low association
with inflammation, while swollen joints were shown to be highly associated with
inflammationasdefinedbyultrasound.
https://doi.org/10.1177%2F1759720X12460116
https://www.the-rheumatologist.org/article/musculoskeletal-ultrasou
nd-a-valuable-tool-for-diagnosing-rheumatic-illnesses/5/?singlepag
e=1

Opticalbiomarkersseeingthatproblematicsynovitis
Validityanddiagnosticperformance of fluorescence
opticalimagingmeasuringsynovitisinhand
osteoarthritis:baseline resultsfromtheNor-Hand
cohort
Maugestenetal.(ArthritisResearch &Therapy 2020)
Department of Rheumatology, Diakonhjemmet Hospital, Oslo
https://doi.org/10.1007/s00256-019-03180-6
Fluorescence optical imaging (FOI) demonstrates enhanced
microcirculation in finger joints as a sign of inflammation. We
wanted to assess the validity and diagnostic performance of FOI
measuring synovitis in persons with hand OA, comparing it with
magneticresonance imaging (MRI)- andultrasound-detected synovitis.
FOI enhancement correlated poorly with synovitis assessed by more
established imaging modalities, questioning the value of FOI for
the evaluation ofsynovitisin hand OA.
Fluorescence opticalimagingfortreatmentmonitoring
inpatientswithearly and activerheumatoidarthritisina
1-year follow-up period
Glimmetal. (ArthritisResearch & Therapy 2019)
Department of Rheumatology and Clinical Immunology, Charité-UniversitätsmedizinBerlin
https://doi.org/10.1186/s13075-019-1989-5
Fluorescence optical imaging (FOI) enables visualization of
inflammation in the hands in rheumatic joint diseases with currently a
lack of long-term follow-up studies. Reduced early enhancement in FOI
phase 1 can be observed in clinically responding and non-responding
early RA patients under treatment. Regarding potential marker
performance, FOI probably shows a reduction of
inflammation more objectively.
Reduction of early enhancement in FOI
(fluorescence optical imaging) phase 1 after 12
months follow-up: a V0: Example with early high
enhancement in phase 1 before ICG flooding in the
fingertips, especially in the wrists, PIPs, and IPs of
both hands. Moderate enhancement in MCP II and
IV of the right hand. V12: High physiological
enhancement in the fingertips in phase 1 after 12
months. No enhancement in the finger and hand
joints. b Example of early enhancement in phase 1
in both hands, especially in MCP II and III ofthe right
hand. High enhancement also in PIPs of both
hands, left wrist, and MCP II and III. Physiological
signal in the fingertips. V12: High physiological
enhancement in the fingertips in phase 1 after 12
months. No significant enhancement in the finger
and hand joints. V0: baseline, V12: follow-up after 12
months

VascularEnvironment inArthritis
Goingwiththeflow:harnessingthe powerofthe
vasculature for targeted therapyinrheumatoidarthritis
Mathieu Ferrari, Shimobi C. Onuoha, Costantino Pitzalis WilliamHarvey Research Institute, BartsandtheLondon SchoolofMedicineandDentistry, Queen Mary University ofLondon
Drug DiscoveryTodayVolume21, Issue 1, January 2016, Pages 172-179
https://doi.org/10.1016/j.drudis.2015.10.014
Although much is still to be learned about the aetiology Rheumatoid
arthritis (RA), a growing body of evidence suggests that an altered
vascular environment is an important aspect of its
pathophysiology. In this context, RA shares many similarities with
cancer, and it is expected that several angiogenic targets in cancer
might be relevant to the treatment of RA. Here, we discuss how these
targets can be combined with advances in drug development to
generate the next generation of RA therapeutics. The therapeutic
possibilities for targeting vascular markersare almost limitless. However,
targeting specific vascular markers for the treatment of RA remains a
relatively unexplored field. More research has been carried out in
the field of cancer.
Whether angiogenesis is a concurrent cause or a consequence
(e.g. Paleolog2009, Konisti et al. 2012 ) during RA onset is still under
debate; however, it is now clear that inflammation and angiogenesis are
two intertwined events that both have a pivotal role in disease
progression andperpetuation
It is now clear that a major paradigm shift is occurring in the treatment of
arthritis. The recent development of small-molecule kinase inhibitors for
the treatment of RA stands to challenge the dominance of biologics as
the first line of care forpatientswho do not respond to methotrexate
Roleof angiogenesis in
rheumatoid arthritis (RA).
Schematic representation of
crosstalkbetween angiogenesis
and inflammation in RA. Local
synovial hyperproliferation
promotesahypoxiccondition
stimulatingangiogenesis, which
exacerbatesthe recruitment of
lymphocytes. Increased
cellularityand cytokine
expression within thesynovium
further stimulate angiogenesisina
positive feedback loop. Blockade
ofthe angiogenic cascade and
pro-inflammatorycytokinescan
stimulate asynergistic antiarthritic
activity. Abbreviations:HIF,
hypoxia-inducible factor;
VEGF(R), vascular endothelia
growth factor (receptor).
Structureof
vasculartargeting
therapeuticsin
developmentfor
treatmentof
rheumatoid arthritis.

invivo VascularImaging forArthritis?
Altered LymphaticVesselAnatomyand Markedly
DiminishedLymphClearance inthe RheumatoidHand
withActive Arthritis
Richard D. Bell etal. (May2020)
https://doi.org/10.1002/art.41311
Assess differences in lymphatic function of RA hands with active synovitis
versus healthy controls (Ctl) via near infrared‐indocyanine green (NIR‐
ICG) imaging. … Most importantly, our findings underscore the potential
contribution of the lymphatic system to the initiation and persistence of
synovial inflammation and highlight, the potential utility of NIR-ICG imaging
as a biomarker of disease. Additional investigations into the mechanisms that
promote or restrict lymphatic flow from synovial tissue have the potential to
revealnewtargetsandimproveclinicaloutcomesforRApatients
Near infrared (NIR) imaging (in this
KITE EMCCD camera) enables
quantification of lymphatic function
following the injection of a
fluorescent tracer that is
incorporated into the interstitial
fluid
Label-free photoacousticimagingofhumanpalmar
vessels:astructuralmorphologicalanalysis
YoshiakiMatsumoto etal. (2018)
https://doi.org/10.1038/s41598-018-19161-z
We analysed the vascular morphology of the palm using a
photoacoustic tomography (PAT) instrument with a hemispherical
detector array. The three-dimensional (3D) morphology at a
resolution of less than 0.5 mm of blood vessels was determined
noninvasively.
Exampleof a blood vessel imageof thepalmtakenat
a wavelength of 795 nmusingthethird prototypeof
ourphotoacoustic imaging system.
Typical examplesof blood vesselswith
different curvatures. Three caseswith
different curvaturesof the common and
proper palmar digital arteriesare shown.

Multispectralimaging forarthritis
Multispectral imagingforpreclinical
assessmentofrheumatoidarthritismodels
Glinton, Sophie L. (2019). UniversityofBirmingham. Ph.D.
https://etheses.bham.ac.uk/id/eprint/9008/
The main aim of this thesis is to determine whether
multispectral imaging of murine arthritis models has the
potential to assess the severity of arthritis symptoms
in vivo in an objective manner. Given that pathology can
influence the optical properties of a tissue, changes may be
detectableinthespectralresponse.
Monte Carlo modelling of reflectance and
transmittance for varying levels of blood volume fraction,
blood oxygen saturation, and water percentage in the mouse
paw tissue demonstrated spectral changes consistent with the
reported/published physiological markers of arthritis.
Subsequent reflectance andtransmittance in vivospectroscopy
of the hind paw successfully detected significant spectral
differences between normal and arthritic mice. Using a novel
non-contact imaging system, multispectral reflectance and
transmittance images were simultaneously collected, enabling
investigation of arthritis symptoms at different anatomical paw
locations.
The results of the experiments are indicative that
multispectral imaging performs well as an assessor of
arthritis for RA models and may outperform existing
techniques. This has implications for better assessment of
preclinical arthritis and hence for better experimental
outcomesandimprovementofanimalwelfare.
To date, to the author’s knowledge, no work has been
published on widefield hyperspectral/
multispectral transillumination of the arthritic
human joint. Tomographic reconstruction of tissue
properties from such images would be inappropriate
given the number of potential sources, but analysis of the
spectroscopic data or topographic tissue property
reconstruction may be able to detect differences in the
arthritic joint (preliminary human spectral data shown in
section 8)

‘Functional’measurements Vibroarthrography
Easiertomeasurethekneeinsteadofhandjoints,butmaybeaglove-type measurementcouldbepossible?Requiresprobably someclever
manufacturing of thegloveto accommodatetheinter-invididualhandsizevariations?
Vibroarthrography usingconvolutional
neuralnetworks
DimitriKraft University ofRostock,Rostock, Germany
and Gerald Bieber
PETRA'20:Proceedings ofthe13th ACMInternational Conference on PErvasiveTechnologies Related to
AssistiveEnvironmentsJune2020 Article No.:58 Pages 1–6
https://doi.org/10.1145/3389189.3397993
Knees, hip, and other human joints generate noise and vibration
while theymove. Thevibration and sound pattern is characteristic
not only for the type of joint but also for the condition. The pattern vary
due to abrasion, damage, injury, and other causes. Therefore, the
vibration and sound analysis, also known as vibroarthrography
(VAG), provides information and possible conclusions about the joint
condition, age and health state. The analysis of the pattern is very
sophisticated and complex and so approaches of machine learning
techniqueswereappliedbefore.
Ingeneral,forbioacoustics,seee.g.
https://doi.org/10.1016/j.cmp
b.2016.03.021
https://doi.org/10.1589/jpts.28.2904 https://doi.org/10.1007/s11517-
018-1785-4
Average maps of the (i) averaged rectified values (ARV,
mm•s-2
),(ii) mean power frequency (MPF,Hz),(iii) variance
of means squared (VoMS, mm-4
•s-8
), (iv) form factor (FF,
a.u.), (v) % of recurrence(%REC), and (vi) % of
determinism (%DET) of the vibroathrographic signals
recorded using eight accelerometers (black dots ) during
activities of daily living (sit to stand, stairs descent and
stairs ascent) among patients with knee osteoarthritis
(n=20) andasymptomatic participants(n=20).
Wirelessmultichannel vibroarthrographic recordingsfor
the assessment of knee osteoarthritisduring three
activities ofdailyliving
https://doi.org/10.1016/j.clinbiomech.2019.11.015
Biomedical Acoustics Special Interest Group (BASIG) is
concerned with the investigation of acoustic and ultrasound wave
interaction with biological materials including soft tissue, bone, and organ
systems and with the measurement, modelling and analysis of sound
generated within thebodyeg lung sounds,speech,circulatorysounds,etc
https://acoustics.ac.uk/sigs/biomedical-acoustics/
Wearableglove-likeMEMSmicrophonearrayforarthritis?
https://www.embedded-computing.com/guest-blogs/mems-packaging-
for-high-volume-products
https://www.techbriefs.com/component/content/article/tb/supple
ments/st/features/technology-leaders/37014

Muscle Activation
quantification
tocomplement
Arthritis
Managament

Daniel Chizhik, UniversityofMaine (2020)
https://digitalcommons.library.umaine.edu/etd/3167/

Alternativeforcomputervision forfingerangleestimation: sEMG
FingerAngleEstimationFrom ArrayEMGSystemUsing
LinearRegressionModelWithIndependentComponent
Analysis SorawitStapornchaisitetal.(2019)
FrontiersinNeurorobotics https://doi.org/10.3389/fnbot.2019.00075
Surface electromyography (EMG) signals from the forearm used in prosthetic
hand and finger control systems require precise anatomy data of finger
muscles that are small and located deep within the forearm. The main
problem of this method is that the signal quality depends on the
placement of EMG sensor, which can significantly affects the accuracy and
precision to estimate joint angles or forces. Moreover, in case of amputees, the
location of finger muscles is unknown and needed to be identified manually for
EMG recording. As a result, most modern prosthetic hands utilize limited
number of muscles with pattern recognition to control finger according to pre-
defined grip which is unable to mimic natural finger motion. To address such
issue, we used array EMG sensors to obtain EMG signals from all possible
positions on the forearm and applied regression method to produce natural
fingermotion.
The result from this study showed that not only array EMG sensors with ICA
significantly improve the quality of signal detected from forearm but also
reduce problems of conventional EMG sensors and consequently
improvetheperformanceofregressionmethodtoimitatenaturalfingermotion.
To have a ground-truth measurement of the finger angles, we used the
convolutional-pose-machines (CPM), a state-of-the-art pose estimator, to
detect the finger position in 2D space. The 21 joints' 2D position from the five
fingers were fed into a hand-finger-model, which was trained using example
imagesfroma Realsensedepthcamera.
In the future, we would like to combine
many sensor signals into two groups of
single joint motion by developing the
MSM model to be able to use multiple
muscles to estimate joint angle. The
machine learning and deep learning also
able to utilize Array EMG system better
than normal linear regression.

sEMGusefulforarthritismanagementtoo #1
Differencesinmuscle activity duringhand-dexteritytasks
betweenwomenwitharthritisand ahealthyreferencegroup
SofiaBrorssonetal.(2014)Health and Welfare, Dala Sports Academy, Dalarna University, Sweden
BMC MusculoskeletalDisordersvol 15,Articlenumber: 154
https://doi.org/10.1186/1471-2474-15-154
Impaired hand function is common in patients with arthritis and it affects
performance of daily activities; thus, hand exercises are recommended. There
is little information on the extent to which the disease affects activation of the
flexor and extensor muscles during these hand-dexterity tasks. The purpose of
this study was to compare muscle activation during such tasks in subjects with
arthritisandinahealthyreferencegroup.
Muscle activation was measured in m. extensor digitorium communis (EDC) and in
m. flexor carpi radialis (FCR) with surface electromyography (sEMG) in women
with rheumatoid arthritis (RA, n = 20), hand osteoarthritis (HOA, n = 16) and in a
healthy reference group (n = 20) during the performance of four daily activity tasks
and four hand exercises. Maximal voluntary isometric contraction (MVIC) was
measured to enable intermuscular comparisons, and muscle activation is presented
as%MVIC.
A strength test performed on the two devices EX-it and Grippit. EX-it is adevice for
evaluation offinger extensionforce(inNewtons,N),whileGrippit measuresgripforce
(Detektor AB, Göteborg, Sweden). The subjects scored their hand function with the
outcome measure Quick Disabilities of Arm, Shoulder, and Hand (Quick
DASH). Women with arthritis tendto use higherlevels of muscle activation
in daily tasks than healthy women, and wrist extensors and flexors appear to be
equally affected. It is important that hand training programs reflect real-life
situations andfocus alsoon extensorstrength.
Grippit
https://doi.org/10.1080/10749357.2016.1168591
https://pubmed.ncbi.nlm.nih.gov/18776605/
Muscle activation wasmeasured withsurface EMG

sEMGusefulforarthritismanagementtoo #2
Impairmentofelectricalactivationof wristflexor andextensor
musclesduringgrippingand functionalactivitiesinthe earlystageof
handosteoarthritis:Across-sectionalstudy NatáliaBarbosaTossinietal.
(2020)Journal ofHandTherapy https://doi.org/10.1016/j.jht.2019.12.010
Thiscross-sectional studydeterminedthat
• Hand osteoarthritisin theinitialstages may contributeto the activationdeficit oftheflexor
andextensormusclesofthewrist.
• Handosteoarthritisintheinitial stageshasanegativeimpacton handfunction.
• There is a functional deficit in this population with no change in maximum grip
strengthintheinitialstagesofthedisease.
The wrist extensor muscles have a fundamental role in the stabilization of the wrist while
performing manual activities. However, it is unknown if the clinical signs of hand
osteoarthritis (HOA)causeimpairmentintheactivationofthesemuscles.
For the evaluation of maximum grip strength, a manual hydraulic dynamometer of the
brand JAMAR was used (Hydraulic Hand Dynamometer – ModelPC-5030J1, Fred
Sammons, Inc, Burr Ridge, IL). We followed the recommendations of the American Society of
Hand Therapists and participants performed three repetitions for 6 s each and 1-min rest
periods.
Electromyography of the extensors and flexors of the wrist was evaluated through the
results shown by the surface electromyography (sEMG). The device Trigno Wireless
System (Delsys Inc, Boston, EUA) was used with asampling frequency of 1200 Hz and three
Trigno Wireless Sensor surface electrodes (Trigno EMG sensor, Delsys Inc, Boston, MA), the
rejection ratehigherthan 80dB.
We can conclude that in the early stages of hand osteoarthritis (HOA), there is
a deficit in the activation of wrist flexor and extensor muscles when performing
manual activities involving pinching and/or gripping with no forearm support.
Furthermore, patients with HOA with no change of grip strength present functional
deficit even in early stages. Those findings are relevant for clinical application;
they show that exercises that activate forearm muscles in functional activities
are also important for clinical protocols of rehabilitation, as well as the ones
that activate intrinsic muscles of the hand. For instance, both are already part of most
protocols. Finally, it is important to preserve grip strength in early degrees of
HOA to avoid disability along the disease progression.

Convergenceherewith“handgripinfitnessandgeneralhealth”?
HandGripTestNormativeDataTo wrap up, decide on adynamometer thatbest
meetsyour requirementsfor testing. We have focused on the Takei grip dynamometers todayfor
thispost, we knowtheyare accurate and reliable and the choice ofmanyprofessional sportsclubs,
academicinstitutionsand the national health service. However, there are other options. If youare
interested takealook at Jamar and Camry.
https://cartwrightfitness.co.uk/hand-grip-test-normative-data/
MagnusMidtbø DestroysNew Grip Strength Test (WORLDRECORD?!) Jujimufu
&Tom,https://www.youtube.com/watch?v=F1S-30foslI → https://gripgenie.com/
https://gripgenie.com/collections/grip-bundles
LifecourseGripStrength|HertfordshireCohortStudy
https://www.mrc.soton.ac.uk/herts/findings/lifecourse-grip-strength/

StimulatingwithEMGelectrodes sense+stimulatewithsamesystem
Seemstobe working for stroke rehab, whataboutarthritis?
Real-timeElectromyography-drivenFunctionalElectrical
StimulationCyclingSystemforChronicStrokeRehabilitation
YuqiFang;SaiChen;XiaojunWang;KenryW.C.Leung;XinWang;Kai-
YuTong
Departmentof BiomedicalEngineering,ChineseUniversityof HongKong
201840thAnnualInternationalConferenceoftheIEEEEngineeringin
MedicineandBiologySociety(EMBC)
https://doi.org/10.1109/EMBC.2018.8512747
Stroke-induced lower extremity dysfunction has become a severe
medical problem nowadays and effective rehabilitation methods are
in great demand. In this work, a new real-time Electromyography-
driven Functional Electrical Stimulation (FES) cycling
system was developed to help chronic stroke patients with lower
limbrehabilitationtraining.
To evaluate the feasibility and effectiveness of this system, 3 chronic
stroke subjects were recruited and each received 20 training
sessions where real-time Electromyography (EMG) was used
to interact with the cycling system. During the training, two typical
metrics, averaged Area Under Torque (AUT) and maximal EMG
amplitude, were adopted to measure the muscle strength
changes of hamstring (HS). The training results showed that the
two measurements of HS both significantly increased, especially the
maximal EMG amplitude in the last trial was twice as much as that in
the first trial, indicating paretic limb strength increment and
functional recovery, which suggested that our system is effective
andhelpfulinthestrokerehabilitation.

Hand Pose Tracking for Clinical Applications

Hand Pose Tracking for Clinical Applications

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Hand Pose Tracking for Clinical Applications

Ähnlich wie Hand Pose Tracking for Clinical Applications (20)

Mehr von PetteriTeikariPhD

Mehr von PetteriTeikariPhD (16)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Hand Pose Tracking for Clinical Applications