Weakly supervised segmentation and Firma convenzione ...
Transcript of Weakly supervised segmentation and Firma convenzione ...
Firma convenzione
Politecnico di Milano e Veneranda Fabbrica
del Duomo di Milano
Aula Magna – Rettorato
Mercoledì 27 maggio 2015
Weakly supervised segmentation and recognition of surgical gestures
Tutor: Elena De Momi, PhD
Co-tutor: Danail Stoyanov, PhD
Co-tutor: Hirenkumar Nakawala, PhD
Student: Beatrice van Amsterdam
850736
Academic Year: 2016-2017
Nome Cognome, assoc.prof. ABC Dept.
IntroductionRobot-Assisted Minimally Invasive Surgery (RAMIS)
Robot-Assisted Minimally Invasive
Surgery (RAMIS)
✓ Video and robot kinematics recoding
✘ Lack of tactile sensation✓✘ Improved visibility (3D)
✓ Improved precision and ergonomic comfort✓ Reduced trauma, pain, recovery time, costs
✘ Extended operation time and learning curve
Minimally Invasive Surgery (MIS)
✓ Reduced trauma, pain, recovery time, costs
✘ Lack of tactile sensation✘ Kinematic restrictions
✘ Reduced visibility (2D)
Open surgery
✓Maximum visibility✓ Tactile sensation✓ No kinematic restrictions✘ Substantial trauma and pain✘ Long recovery time
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
IntroductionBackground clinical problem
Surgical technical errors are the mostcommon reason (30%) for post-surgicalcomplications including re-operation andre-admission. (Stahel, 2014)
(Birkmeyer, 2013)
The issue of surgical errors has becomeparticularly relevant with the advent ofnew surgical techniques that offer greatadvantages to the patient but require ahigh level of visuomotor skill from thesurgeon, such as MIS and RAMIS. (White,
2015)
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
IntroductionAction recognition for Computer Assisted Interventions
❖ Extended learning curve Computational models for surgical skill assessment
❖ Critical and time-consuming tasks Intra-operative assistance and automation
Surgery involves dexterous human motion. The eventual goal for studying surgical
motion could improve safety and effectiveness of surgical patient care (Gao, 2014) .
Beatrice van Amsterdam
ACTION RECOGNITION
Understand where the surgeonneeds improvement
Short segments are easier toautomate
Nome Cognome, assoc.prof. ABC Dept.
IntroductionAction granularity
Beatrice van Amsterdam
Procedure
Phase
Activity
Surgeme
Dexeme
E.g. Suturing
E.g. Closure
E.g. Lumbar Disc Herniation
E.g. Grasping the needle
E.g. Turning left
(Lalys, 2014)
Nome Cognome, assoc.prof. ABC Dept.
Related work
Supervised Approaches
• Hidden Markov Models (Tao, 2012)
• Conditional Random Fields (Tao, 2013)
• Deep Learning (Lea, 2017)
Unsupervised Approaches
• Gaussian Mixture Models (Murali, 2016)
• Hidden Markov Models unsup. (Tanwani, 2016)
• Bottom-up clustering (Fard, 2016)
Weakly supervised Approaches
• Video retrieval (Quellec, 2014)
• CNN (Wang, 2017)
• Ordering constrained clustering (Bojanowski, 2014)
Beatrice van Amsterdam
✓ Good results✘ Manual labelling (burden, subjective)✘ Fixed set of classes
✓ No manual labelling
✘ Poorer performance✓ Classes learnt from data
✓ Better performance than unsupervised methods✓ ✘ Minor manual labelling✘ Fixed set of classes
Nome Cognome, assoc.prof. ABC Dept.
Aim of the work
New approach: employ weak supervision to optimize
unsupervised algorithm initialization
Beatrice van Amsterdam
Many unsupervised algorithms rely on iterative schemes: initialization problem
Expected outcome: improved alignment between
predicted and ground truth segmentation
Nome Cognome, assoc.prof. ABC Dept.
Materials and MethodsThe JIGSAWS dataset
Beatrice van AmsterdamBeatrice van Amsterdam
JIGSAWS (Gao, 2014) is a public dataset featuring suturing demonstrations collected fromeight surgeons with different skill level (Expert, Intermediate, Novice) using the da VinciSurgical System.
Cartesian positions (3 variables)Rotation matrix (9 variables)Linear velocities (3 variables)
Angular velocities (3 variables)Gripper angle (1 variable)
The motion of each manipulator isdescribed by a local frame attached at itsend-effector using 19 kinematic variables:
Expert: > 100 training hours Novice: < 10 training hours
Nome Cognome, assoc.prof. ABC Dept.
Materials and MethodsPre-processing
Rotation matrix converted into
quaternion
Euclidean distance signals
computation
Low pass filter
fc = 1.5 Hz
(Despinoy, 2016)
Normalization to zero mean and unit variance
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
Materials and MethodsSimultaneous action segmentation and recognition
After model training, each sample is assigned to itsmost likely mixture component, i.e. its most likelysurgeme label.
Each demonstration 𝒙 𝑡 ∈ ℝ𝑝 can be treated as anoisy realization of a switched linear dynamicalsystem. Each regime 𝑨𝒌 corresponds to a surgeme.
which is equivalent to solving multiple linearregression problems, when L=1 (Moldovan, 2015).
𝒙 𝑡 + 1 = 𝑨𝒌𝒙 𝑡 + 𝒘 𝑡 ,
𝑨𝒌 ∈ {𝑨𝟏, … , 𝑨𝑵}, 𝑨𝒌 ∈ ℝ𝑝×𝑝
argmin𝑨𝒌
𝑨𝒌𝑿𝒕 − 𝑿𝒕+𝟏 ,
𝑿𝒕 = [𝒙 𝑡 ,… , 𝒙 𝑇 ] ∈ ℝ𝑝×𝑇
Action recognition can then be performed byfitting a Gaussian Mixture Model (GMM) to theaugmented state 𝒏 𝑡 ,
𝒏 𝑡 = 𝒙 𝑡 , 𝒙 𝑡 + 1 ,… , 𝒙 𝑡 + 𝐿 ,
𝒏 𝑡 ∈ ℝ𝑝𝐿
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
Materials and MethodsGround truth segmentation redefinition: local linearity
Beatrice van AmsterdamBeatrice van Amsterdam
𝐿 = 3 (Murali, 2016) 𝒏 𝑡 = 𝒙 𝑡 , … , 𝒙 𝑡 + 𝐿 = 𝒙 𝑡 , 𝒙 𝑡 + 1 , 𝒙 𝑡 + 2 , 𝒙 𝑡 + 3
𝒙 𝑡𝒙 𝑡 + 1𝒙 𝑡 + 2𝒙 𝑡 + 3
𝒏 𝑡
Nome Cognome, assoc.prof. ABC Dept.
Materials and MethodsGround truth segmentation redefinition: action dictionary
L1: positioning and pushing needle through the tissue
L2: reaching for the needle with the left hand
L3: pulling suture with the left hand
L4: transferring needle from left to right
L5: extracting suture from the tissue with the left hand
L6: reaching for the needle with the right hand;
L7: transferring needle from right to left
L8: using right hand to help tighten suture
L9: dropping suture
L10: moving to end points
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
Materials and MethodsWeakly supervised initialization
Beatrice van Amsterdam
GMM0
w
Nome Cognome, assoc.prof. ABC Dept.
Materials and MethodsOffline segmentation
Beatrice van Amsterdam
w
w
Nome Cognome, assoc.prof. ABC Dept.
Evaluation protocolExtrinsic evaluation metrics
𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚 =𝑇𝑃
𝑇𝑃 + 𝐹𝑃 + 𝐹𝑁
𝑹𝒆𝒄𝒂𝒍𝒍 =𝑇𝑃
𝑇𝑃 + 𝐹𝑁
𝑷𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 =𝑇𝑃
𝑇𝑃 + 𝐹𝑃
𝑭 − 𝒔𝒄𝒐𝒓𝒆 =2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙
𝑵𝒐𝒓𝒎𝒂𝒍𝒊𝒛𝒆𝒅𝑴𝒖𝒕𝒖𝒂𝒍 𝑰𝒏𝒇𝒐𝒓𝒎𝒂𝒕𝒊𝒐𝒏 (𝑵𝑴𝑰) =𝐼 𝛾, 𝜏
𝐻 𝛾 𝐻 𝜏
𝑹𝒆𝒄𝒂𝒍𝒍 𝑴𝒂𝒄𝒓𝒐𝑾𝒆𝒊𝒈𝒉𝒕𝒆𝒅 (𝑹𝑴𝑾)
𝑷𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏 𝑴𝒂𝒄𝒓𝒐𝑾𝒆𝒊𝒈𝒉𝒕𝒆𝒅 (𝑷𝑴𝑾)
𝑭 − 𝒔𝒄𝒐𝒓𝒆𝑴𝒂𝒄𝒓𝒐𝑾𝒆𝒊𝒈𝒉𝒕𝒆𝒅 (𝑭𝑴𝑾)
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
Evaluation protocolIntrinsic evaluation metrics
𝑺𝒊𝒍𝒉𝒐𝒖𝒆𝒕𝒕𝒆 𝑰𝒏𝒅𝒆𝒙 (𝑺𝑰) = 𝑚𝑒𝑎𝑛 𝒔𝑺𝑰 , 𝒔𝑺𝑰 =𝑏−𝑎
max(𝑎,𝑏), 0 < 𝒔𝑺𝑰 < 1
𝑹𝒆𝒍𝒂𝒕𝒊𝒗𝒆 𝑺𝒊𝒍𝒉𝒐𝒖𝒆𝒕𝒕𝒆 𝑰𝒏𝒅𝒆𝒙 (𝑹𝑺𝑰) =𝑆𝐼
Ground Truth SI
✕
✕
✕
𝑎
𝑏
𝒔
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
Experiments and ResultsExperimental protocol
Beatrice van Amsterdam
❖ Ground truth redefinition
❖ Initialization technique
❖ Feature selection
❖ Increased data variability
Experts only
Full dataset
Nome Cognome, assoc.prof. ABC Dept.
Experiments and ResultsGround truth redefinition (experts)
*p<0.05 McNemar’s test
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
Experiments and ResultsGround truth redefinition (experts)
*p<0.05 𝜒2 test for independent proportions
Beatrice van Amsterdam
Proposed annotation
Original annotation
Nome Cognome, assoc.prof. ABC Dept.
Experiments and ResultsInitialization technique (experts)
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
Experiments and ResultsFeature selection (experts)
McNemar’s test
Beatrice van Amsterdam
Nome Cognome, assoc.prof. ABC Dept.
Experiments and ResultsFeature selection (experts)
Beatrice van Amsterdam
output
GT
GToutput
Nome Cognome, assoc.prof. ABC Dept.
Experiments and ResultsExtended dataset
Beatrice van Amsterdam
< (Lea, 2017) supervised (85% Accuracy)< (Fard, 2016) unsupervised (0.78 Recall, 0.74 Precision)> (Murali, 2016) unsupervised GMM-based (0.31 NMI)
Nome Cognome, assoc.prof. ABC Dept.
Conclusion and future work
Experimental results on real surgical kinematic trajectories during a training exerciseconfirm that weakly supervised initialization signicantly outperforms standard task-agnostic initialization methods, leading to 13% improvement of NMI.
Automatic recognition of surgical gestures is an important preliminary step in thedevelopment of computational models for surgical skill assessment as well as intra-operative assistance and automation algorithms.
Future work:
• Including contextual and semantic information from the video data.
• Extending the kinematic data with the trajectories of the da Vinci arm joints.
• Introducing transition probabilities between different actions (HMM).
Beatrice van Amsterdam