Intelligent Systems July 06

www.haptica.com | tel. 617 342 7270 | email. [email protected]

Efficient rigid body motion

tracking with applications to

human psychomotor performance assessment

“Surgeons who trained on simulations performed the operation 29% faster and made 7 times fewer errors”. Annals of Surgery 2002

Gerard LaceyComputer Vision and Robotics Group

Dept. of Computer ScienceTrinity College Dublin

Haptica ProMIS May2006

Overview of Talk

• Context for research – Haptica Ltd

• Laparoscopic Surgery Intro

• Solutions to Surgical Training

• Computer vision tracking

• Live multimedia annotation of training video

• The results

• The current position

• The Future


Inspiration for ProMIS

• Previous Research• Mobile Robotics for Frail Visually

Impaired Elderly

• Bayesian Belief Networks for multi-sensor classification / context sensitive reasoning

• Haptica Ltd formed in 2000

• Focus on human performance enhancement in critical tasks• Guidance for mobility

• Computer Assisted Docking

• Fraud detection in credit card payments

• Semantic Web search of patent databases

• Laparoscopic Surgery Training


Laparoscopic Surgery

• Laparoscopy – Minimally Invasive Surgery• Surgeons operate with long narrow instruments

• Watch their movements on TV monitors

• Rapidly growing due to patient demand and cost savings in reduced patient stays

• Key Challenges• Loss of dexterity

• Fulcrum effect

• Loss of sense of touch

• Slower than conventional surgery

• Environment for learning• Apprentice model

• See one, do one, teach one

• Patients used for learning

• No objective assessment


Current Training Methods

• Current Training Tools• Patients

• Physical simulators - “Box Trainers”

• Virtual Reality and Full Haptic Simulators

Physical Sim VR Sim

System behaves like abdomen

Realistic tactile feedback

Realistic visual feedback

Use the real tools

Team approach to procedure

Simulate specific surgical procedure

Assess Trainee’s performance

Use with computer based training

Playback a video of their operation

Simulate affect on of errors

Simulate unusual pathologies


Objective Skills Measurement in Medicine

• Human observation of bench tests

• Resnick – U Toronto (OSATS)

• Track hand movements to measure surgical skills

• Darzi – Imperial College (ICSAD)

• Linking skills to patient outcomes

• VR to OR trials

• Key understanding of the skills to be learned

• Cognitive X% + Affective Y% + Psychomotor Skill Z%

• Surgery mainly 75% X & 25% Z with Z higher in minimally invasive surgery


Key Skills in surgery

• Key indicators of skill acquisition• The objective is Proficiency

• The time proxy – How fast can it be done?

• The repetitions proxy – How many times has it been done?

• The errors proxy – have any of these listed errors occurred while the task was being completed?

• Surgical Finesse – smooth movement / economy of movement• Path of surgical instruments in space

Novice Expert


Measuring Economy of Movement

Stereo Cameras Track Instruments


Design of the Stereo system

• Background simplification

• Camera calibration – essential step

• Tracking algorithm• Top 1/3 of the image

• Instrument identification

• Image de-warping

• Icon tracking

• Complications• Multiple instruments –

• Occlusion

• Correspondence disambiguation


Vision Systems Design

• Off the shelf web cams (firewire)

• Calibration of cameras to sub pixel accuracy

• High frame rate (15fps or higher)

• Task trays manufactured to high accuracy

• Task trays rigidly mounted to high accuracy

• Key design factor in enabling low cost Augmented Reality

Bile ductAR


The Augmented Reality system

• Aligning parallel 3D Universe’s• Vision System reference frame

• Graphics Engine reference frame

• Mechanical mounting reference frame

• Registration• Optical Distortion

• De-warping of images

• Occlusion masks (instruments and objects)• Real-time tracking and AR display (on a laptop)

• down and dirty in direct x / direct show


Augmented Reality Concept

• Annotate live video with Computer Mediated Learning content • Teaching Instructions

• Feedback on Progress

• Hybrid tasks

• Adverse Events

ARDemo


Results

• ProMIS Construct Validity Trials• RCSI / AMNCH

• Emory University

• Imperial College

• Very high ability to discriminate novice from expert using economy of movement• Papers published in surgical fora

• Other Validations• European IST prize

• Irish Software Association Technical Innovation Prize

• Society of Laparoendoscopic Surgeons Innovation of the Year award

• Irish Design Institute “Highly Commended Design”


Idea refinement – The importance of key reference customers

• Concept to product • March 02 to December 03

• Patent May 02

• Basic concept – cardboard box

• Stage Gate™ type process

• Multiple Prototypes • with RCSI, Emory (US), Imperial (UK)

• Production prototypes June 03

• Production process set-up Oct 03

• First sales to RCSI December 03


Current Position

• ProMIS 2 Launched October 2005• Hand Tracking for “hand assisted” Procedures

• Model abdominal organs

• Full procedure training for Colectomy

• 8 cameras, tilt-sensor, animatronics

• Still runs on a laptop…

• 7 Papers validating ProMIS published at most recent surgical congress


ProMIS 2.0


Future of Augmented Reality and Video Indexation

• Current Projects• Illicit Content detection

• Colonoscopy: Automated Reporting tools - tagging

• Colonoscopy: Automated cancer detection

• Hand washing quality assessment

• Ground Collision Warning System for Aircraft

Intelligent Systems July 06

Documents

Transcript of Intelligent Systems July 06