Post on 26-Dec-2015
Video Summarization of
Key EventsStage I - The Critical View
Michael A. Grasso, MD, PhDUniversity of Maryland School of
MedicineUMBC Computer Science
MichaelGrasso.com
Abstract
Laparoscopic surgery is a minimally invasive technique with unique training requirements. Video-assisted evaluation is one method that surgical residents can use to demonstrate competence. Automated video summarization can increase the efficiency of evaluations by directing the senior surgeon to key portions of a surgical procedure. We are using image classification techniques to segment videos of laparoscopic cholecystectomies to assist with surgical training and evaluation.
Overview
Background Laparoscopic Surgery Image Classification
Methods Discussion
Laparoscopic Surgery
Minimally Invasive Surgery. First performed in 1987. Used in many surgical procedures.
Gall bladder removal (cholecystectomy).
Esophageal surgery (fundoplication). Colon surgery (colectomy). Others.
Laparoscopic Approach Narrow tubes (trocars)
are inserted into the abdomen through small incisions.
www.fda.gov
Laparoscopic Procedure Camera is passed
through trocar. Procedure is often
videotaped. Carbon dioxide is
infused through trocar.
Instruments are passed through the trocars to cut, manipulate, and sew.
Laparoscopic Aftercare Compared with an
open procedure. Smaller scars. Reduced pain. Quicker recovery.
http://www.nlm.nih.gov/medlineplus/ency/presentations/100166_1.htm
Technical Challenges
Access limited to small incisions. Long instruments with only the tips
visible. Two-dimensional video. Limited tactile feedback.
British Journal of Surgery. 2004 Dec;91(12):1549-1558
Laparoscopic Training
Traditional apprenticeship model. Acquire skills during actual procedures. Not sufficient for laparoscopic skills.
Other methods. Box trainer with animal or synthetic
models. Virtual reality simulator. Video-based assessment.
Assessment of Skills
Trainee must demonstrate competency.
Evaluation by a senior surgeon. Direct observation of the trainee. Video-based assessment.
Question: Can we organize video in order to assist in video-based assessment?
American Journal of Surgery. 1991 Mar;161(3):399-403
Objective
Identity key portions of surgical procedure to aid in video-based assessment.
Stage I is to identify the "critical view".
Video
Segments
Frames
Overview
Background Laparoscopic Surgery Image Classification
Methods Discussion
Summary: Organize surgical video to make it easier for expert to review.
The Critical View
Helps ensure that the anatomy has been properly identified.
Occurs after dissecting anatomy. Occurs before clipping the cystic
artery and cystic duct.
The Critical View
Cystic artery
Liver
Cystic duct
Fundus
Netter's Atlas of Human Anatomy
The Critical View
Image Classification - Human
Features a person might use. Spectral features.
Tonal variations. Textural features.
Spatial distribution of tonal variations. Contextual features.
Features from surrounding areas.
Image Classification - Computed
Features extracted from image. Spectral features.
Distribution, size, width. Textural features.
Homogeneity, contrast, correlation. Similarity/distance metrics.
Jaccard coefficient, Jeffrey divergence.Journal of WSCG. 2003; 11(1):269-273
IEEE Transaction on Systems, Man, and Cybernetics. 1973 Nov; 3(6):610-621
Color Histogram Red, green, blue, or gray.
Count number of pixels for each tone. One 28 set for an 8-bit image for each color. Does not vary with translation and rotation. Ignores shape and texture.
4x4 image. 4 gray tones. H = {5, 4, 5, 2}
0 0 1 1
0 0 1 1
0 2 2 2
2 2 3 3
Binary Histogram
Quantize values for each tone to 0 or 1.
Background color given less weight. Subtle changes given more weight.
HB = {1, 0, 1, 1}
0 0 0 0
0 0 0 0
0 2 2 2
2 2 3 3
3D Histogram
Distribution within a 3D color-space. 3D color space (red, green, blue). Used in object recognition & image
retrieval. n3 entries, where n = number of tones.
Example. Quantized to 3 tones
for each color.
Spatial-Dependency Matrix Co-occurrence
matrix. Co-occurring values
(0o, 45o, 90o, 135o). Four 28 x 28
matrices for 8-bit image.
Co-occurring Bits
0 1 2 3Refe
ren
ce B
its
0 4 2 1 01 2 4 0 02 1 0 6 13 0 0 1 2
0 0 1 1
0 0 1 1
0 2 2 2
2 2 3 3
135o 90o 45o
0o Ref 0o
45o 90o 135o
M0 =
Additional Spectral Features
Location of the distribution. Mean = Σ (bin*freq) / Σ (freq). Mode = bin of the max freq.
Size of the distribution. Standard deviation.
Width of the distribution. Max(bin) - Min(bin).
Additional Textural Features
Homogeneity. Number of tone transitions.
Contrast. Amount of local variation.
Correlation. Measure of linear dependencies.
Similarity/Distance Metrics
Jaccard Coefficient. Similarity of two sample sets.
|A B| / |A B| Two binary sets.
M11 / (M01 + M10 + M11)
Jeffrey Divergence. Distance between two vector spaces.
Σ (xi log(xi/avgi) + yi log(yi/avgi))n
i=1
Other Distance Metrics
City Block or Manhattan Distance. Euclidean Distance. Chi-Square. Canberra Distance.
Proceedings ACM SAC. 2008;:1225-1230
Related Efforts - Hysteroscopy Use Jeffrey divergence on color
histogram to identify segments. Relevant segments based on image
redundancy. No understanding
of the content of each segment.
Proceedings 27th IEEE-EMBS. 2005;:5680-5683
Mayo Clinic
Related Efforts - Echocardiogram
Use cosine similarity and edge change ratio to identify video segments.
State-based modeling. Identify states in each
video segment. Diastole (resting). Systole (contracting).
IEEE Transaction on Information Technology in Biomedicine. 2008 May;12(3):366-376
Medline Plus
Overview
Background Laparoscopic Surgery Image Classification
Methods Discussion
Summary: Spectral and textural features compared with similarity metrics.
Methods
Our objective. Identity key portions of surgical
procedure to aid in video-based assessment.
Stage I is to identify the "critical view".
Video
Segments
Frames
Tools
FFmpeg http://ffmpeg.mplayerhq.hu/ Extract JPEG images.
ImageJ http://rsbweb.nih.gov/ij/ Macros and Java plugins.
Work Plan
Identify videos for analysis. Convert videos to JPG. Evaluate ability to identify critical view.
Color histogram. Binary histogram. 3D histogram. Spatial-dependency matrix. Jaccard coefficient, Jeffrey divergence.
Algorithm
Feature ExtractionImageJ Color Histograms
Binary Histograms3D Histograms
Spatial-Dependency Matrices
Similarity Metric
Critical View?
Critical View
Random Image
Image ExtractionFFmpeg
Overview Background
Laparoscopic Surgery Image Classification
Methods Discussion
Summary: Attempt to identify the critical view by comparing image features with similarity metrics.
Discussion
Color and binary histograms do not correlate with the critical view. They do, however, predict when we
are in the abdomen. Currently working on 3D histograms
and spatial-dependency matrices. NIH grant application under
development.
Challenges
Live tissue (vs. solid objects). Deformable. Normal variation. Disease states.
May need to consider. Temporal information. Relevant clinical data of the patient. Critical view "rectangle" (contextual).
Summary
We are comparing image features with similarity metrics to identify the critical view.
This is a first step in automated video summarization, to help with video-assisted evaluation of laparoscopic surgery.
Thank You