Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST...
-
Upload
lorin-nelson -
Category
Documents
-
view
215 -
download
0
Transcript of Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST...
Stereoscopic Video Overlay with Deformable Registration
Balazs VagvolgyiProf. Gregory Hager
CISST ERC
Dr. David Yuh, M.D.Department of Surgery
Johns Hopkins University
The CASA Project
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Today’s Surgical Assistant: A Simple Information Channel
The CASA Project
Stereo surface tracking
Stereo tool tracking
Virtual fixtures with
da Vinci Robot
Task graph execution system
HMM-based Intent Recognition
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Information Fusion with
da Vinci Display
Ultrasound
Capabilities of a Context-Aware Surgical Assistant (CASA)
Tissue Classification
PreoperativeImagery
The CASA Project
Stereo surface tracking
Stereo tool tracking
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Information Fusion with
da Vinci Display
Developing a Context-Aware Surgical Assistant (CASA)
PreoperativeImagery
Information Overlay
• Problem setting:– Given pre-operative scan
data from a suitable imagingmodality
– Video sequence from a stereo endoscope
• Add value– Overlay underlying anatomy on the stereo video
stream (x-ray vision)
– Include annotations or other information tied to imagery
Key Problem: Nonrigid registration of organ surface to data
Inputs: What Do We Know?
1. Pre-operative 3D model- most probably volumetric- only a portion of it will be visible on the endoscope- anatomy will be deformed during the surgical procedure
2. Camera system properties can be measured- optical & stereo calibration- local brightness/contrast/color response
3. Stereo image stream- 3D surface can be reconstructed- texture information
4. A guesstimate of model–endoscope 3D relationship- We can guess where to start searching [i.e. patient position]
Outputs: What Do We Generate?
1. Position of 3D model registered to stereo image
2. Model deformed to the current shape of anatomy
3. Rendering a synthetic 3D view on the stereo stream
4. Everything done real-time
Original Image Stereo Data Deformed Mesh
2D 3D
All this in a flow chart
Stereo imagepre-processing
Building andoptimizing
disparity map
DeformableRegistration to
3D surface
3D texturetracking
Recognizingdeformations
optical parameters
stereo video stream
Imageoverlay
disparity
3D data
image data
parameters3D model
Classical Stereo Vision: The Problem
• Blocks of each image are compared using SAD
• Optimization for each block independently on entire depth range
+ Very fast implementation (GPU)
¬ Lousy results
Small Vision Systemfrom Videre Design
(w/o structured light):
• Input images downsized to several scale levels (½, ¼, …)• Each scale processed with the same algorithm
– Propagate coarse search results to the finer scale
+ Quality of disparity map is better + Even faster than single scale computation¬ Requires
structured light
Solution #1: Lighting and Multi-Scale
SVL implementation(using structured light):
• Solve a (spatially) global optimization with regularization
– O(D) = min SAD(D) + Smooth(D)
• GLOBAL optimum found in polynomial time
Solution #2: Dynamic Programming
1. Defining the recursive cost function
2. Memoization
3. Finding lowest cost path, which is the disparity map (DM in red)
SmoothnessError
Solution #2: Dynamic Programming
Dynamic Programming on Images
• Minor issue: previous approach applies to scanline
• Approximate DP applied to entire image
- 3D disparity space (D):
- Cost function (C):
- Memoization (P):
Dynamic Programming: Results
Dynamic Programming: In Vivo Results
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Stereo recordings from the da Vinci robot Focal length of ~ 700 pixels ~5mm baseline Distance to surface of 55mm to 154mm.
Raw Disparity Map Textured 3D Model
Surface to 3D Model Registration
• Inputs:– point cloud from the stereo surface modeler– point cloud generated from a model or volume image
• Outputs:- transformation to register the 3D model to the 3D surface
QuickTime™ and a decompressor
are needed to see this picture.
Results: Rigid Registration
Complete system (stereoplus registration) operatesat 5 frames/second
QuickTime™ and a decompressor
are needed to see this picture.
Current algorithm usesIPC with modificationsto account for occlusionsdue to viewpoint (z-buffer)
From Rigid to Deformable
• Calculate residual errors in z direction
• Define a spring-mass system
• Perform local gradient descent
Deformable Registration Results
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Final registration error of < 1mm exceptfor the area where the tool enters the image
Coming in CASA
The Language of Surgery
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Tool Tracking
Tissue Surface Classification
Thank you!
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.