Knowledge Systems Lab JN 9/13/2015 An Advanced User Interface for Pattern Recognition in Medical...
-
Upload
gillian-pitts -
Category
Documents
-
view
213 -
download
0
Transcript of Knowledge Systems Lab JN 9/13/2015 An Advanced User Interface for Pattern Recognition in Medical...
JN 04/19/23
Knowledge Systems Lab
An Advanced User Interface for Pattern Recognition in Medical Imagery:
Interactive Learning, Contextual Zooming, and Gesture Recognition
Joshua R. NewKnowledge Systems Laboratory
Jacksonville State University
JN 04/19/23
Knowledge Systems Lab
Outline
• Introduction• Techniques: Segmentation, Magnification,
Exploration
• Solutions: – Interactive Learning– Contextual Zooming– Gesture Recognition
• Conclusions
JN 04/19/23
Knowledge Systems Lab
Introduction
Medical imagery… • Consists of millions of images produced
annually which doctors must gather and analyze
• Entails several modalities for each patient, such as MRI, CT, and PET
Refine techniques for facilitating comprehension of this data
JN 04/19/23
Knowledge Systems Lab
Outline
• Introduction• Techniques: Segmentation,
Magnification, Exploration
• Solutions: – Interactive Learning– Contextual Zooming– Gesture Recognition
• Conclusions
JN 04/19/23
Knowledge Systems Lab
Techniques
• Common techniques for facilitating data comprehension:– Segmentation – Labeling of images– Magnification – Precision viewing– Exploration – Interacting intuitively with
complex, 3D data
JN 04/19/23
Knowledge Systems Lab
Why Segmentation?
• Doctors and radiologists:– Spend several hours daily analyzing
patient images (ie. MRI scans of the brain)– Search for patterns in images that are
standard and well-known to doctors
• Why not have the doctor teach the computer to find these patterns in the images?
JN 04/19/23
Knowledge Systems Lab
Why Magnification?
• Doctors and radiologists:– Must be able to precisely view and select
regions/pixels of the image to train the computer
– Can easily lose where they are looking in the image when using magnification
• Why not use visualization techniques to preserve context while allowing precise selections?
JN 04/19/23
Knowledge Systems Lab
Why Exploration?
• Doctors and radiologists:– Need to intuitively interact with the system
to maximize task performance– Need to perform this interaction while
being unencumbered
• Why not use vision-based recognition to allow interaction with the data?
JN 04/19/23
Knowledge Systems Lab
Outline
• Introduction• Techniques: Segmentation, Magnification,
Exploration
• Solutions: – Interactive Learning– Contextual Zooming– Gesture Recognition
• Conclusions
JN 04/19/23
Knowledge Systems Lab
Problems & Solutions
• Problem #1: Segmentation• Solution #1: Interactive Learning
• Problem #2: Magnification• Solution #2: Contextual Zoom
• Problem #3: Exploration• Solution #3: Gesture Recognition
JN 04/19/23
Knowledge Systems Lab
Platform
• Med-LIFE:– “L”earning of MRI image patterns– “I”mage “F”usion of multiple MRI images– “E”xploration of the fusion and learning
results in an intuitive 3D environment
• Images used from “The Whole Brain Atlas”– http://www.med.harvard.edu/AANLIB/home.html
JN 04/19/23
Knowledge Systems Lab
Outline
• Introduction• Techniques: Segmentation, Magnification,
Exploration
• Solutions: – Interactive Learning– Contextual Zooming– Gesture Recognition
• Conclusions
JN 04/19/23
Knowledge Systems Lab
Simplified Fuzzy ARTMAP
• Simplified Fuzzy ARTMAP (SFAM)– An AI neural network
(NN) system– Capable of online,
incremental learning– Takes seconds for
tasks that take backpropagation NNs days or weeks to perform
JN 04/19/23
Knowledge Systems Lab
Vector-based Learning
• Two “vectors” are sent to this system for learning:– Input feature vector provides the data from
which SFAM can learn– ‘Teacher’ signal indicates whether that
vector is an example or counterexample
JN 04/19/23
Knowledge Systems Lab
Learning Visualization
• Vector-based graphic visualization of learning
Array of Pixel Values
x
y
Category 1 - 2 members
Category 2 - 1 member
Category 4 - 3 members
0.30 0.45
JN 04/19/23
Knowledge Systems Lab
Varying Vigilance
• Only one tunable parameter – vigilance– Vigilance can be set from 0 to 1 and corresponds to the
generality by which things are classified(ie. vig=0.3=>human, vig=0.6=>male, 0.9=>Joshua New)
0.675 0.75 0.825
JN 04/19/23
Knowledge Systems Lab
Input Order Dependence
• SFAM is sensitive to the order of the inputs
x
y
Category 1 - 2 members
Category 2 - 1 member
Category 4 - 3 members
Vector 3
Vector 1
Vector 2
JN 04/19/23
Knowledge Systems Lab
Heterogeneous Network
• Voting scheme of 5 Heterogeneous SFAM networks to overcome vigilance and input order dependence– 3 networks: random input order, set vigilance
– 2 networks: 3rd network order, vigilance ± 10%
JN 04/19/23
Knowledge Systems Lab
Segmentation Results
Threshold results
Overlay results
Trans-slice results
JN 04/19/23
Knowledge Systems Lab
Segmentation Solution
• Doctors and radiologists:– Spend several hours daily analyzing patient
images (ie. MRI scans of the brain)– Search for patterns in images that are standard
and well-known to doctors
• Solution:– Doctors and radiologists can teach the
computer to recognize abnormal brain tissue– They can refine the learning systems results
interactively
JN 04/19/23
Knowledge Systems Lab
Outline
• Introduction• Techniques: Segmentation, Magnification,
Exploration
• Solutions: – Interactive Learning– Contextual Zooming– Gesture Recognition
• Conclusions
JN 04/19/23
Knowledge Systems Lab
Research & Business
• Carpendale PhD Thesis– Elastic Presentation Space – rubber sheet
images via mathematical constructs
• IDELIX (www.idelix.com)
– Pliable Display Technology – software development kit (SDK) product
– Boeing: 20% increase in productivity
JN 04/19/23
Knowledge Systems Lab
Magnification Solution
• Doctors and radiologists:– Must be able to precisely view and select
regions/pixels of the image to train the computer– Can easily lose where they are looking in the
image when using magnification• Solution
– They can precisely select targets/non-targets– They can zoom for precision while maintaining
context of the entire image– The interface facilitates task performance
through interactive display of segmentation results
JN 04/19/23
Knowledge Systems Lab
Outline
• Introduction• Techniques: Segmentation, Magnification,
Exploration
• Solutions: – Interactive Learning– Contextual Zooming– Gesture Recognition
• Conclusions
JN 04/19/23
Knowledge Systems Lab
Motivation
• Gesturing is a natural form of communication:– Gesture naturally while talking– Babies gesture before they can talk
• Interaction problems with the mouse:– Have to locate cursor– Hard for some to control (Parkinsons or
people on a train)– Limited forms of input from the mouse
JN 04/19/23
Knowledge Systems Lab
Motivation
• Problems with the Virtual Reality Glove as a gesture recognition device:– Reliability– Always connected – Encumbrance
JN 04/19/23
Knowledge Systems Lab
System Diagram
StandardWeb Camera
Rendering
User InterfaceDisplay
Han
d
Movem
ent
User
Gesture Recognition
System
ImageCapture
Update Object
Image Input
JN 04/19/23
Knowledge Systems Lab
System Performance
• System:• OpenCV and IPL libraries (from Intel)
• Input:• 640x480 video image• Hand calibration measure
• Output:• Rough estimate of centroid• Refined estimate of centroid• Number of fingers being held up• Manipulation of 3D skull in QT interface in
response to gesturing
JN 04/19/23
Knowledge Systems Lab
Calibration Measure
• Max hand size in x and y orientation (number of pixels in 640x480 image)
JN 04/19/23
Knowledge Systems Lab
Saturation Extraction
Saturation Channel Extraction (HSL space):
Original ImageOriginal Image Hue
Lightness
Saturation
JN 04/19/23
Knowledge Systems Lab
Gesture Recognition Pipeline
a) 0th moment of an image:
b) 1st moment for x and y of an image, respectively:
c) 2nd moment for x and y of an image, respectively:
d) Orientation ofimage major axis:
2
arctan2
00
022
00
20
00
112
cc
cc
yM
Mx
M
M
yxM
M
),(220 yxIxM
),(00 yxIM
),(10 yxIxM ),(01 yxIyM
),(202 yxIyM
JN 04/19/23
Knowledge Systems Lab
Gesture Recognition Pipeline
)(*19.0 HandSizeYHandSizeXRadius
• The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses
• A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance)
• Total fingers is number of fingers minus 1 for the hand itself
JN 04/19/23
Knowledge Systems Lab
Interaction Mapping
Gesture to Interaction Mapping
Number of Fingers:
2 – Roll Left3 – Roll Right
4 – Zoom In5 – Zoom Out
JN 04/19/23
Knowledge Systems Lab
Exploration Solution
• Doctors and radiologists:– Need to intuitively interact with the system to
maximize task performance– Need to perform this interaction while being
unencumbered
• Solution– Can use intuitive gesturing to interact with
complex, 3D data– Can interact by simply moving their hand in
front of a camera, requiring no physical device manipulation
JN 04/19/23
Knowledge Systems Lab
Outline
• Introduction• Techniques: Segmentation, Magnification,
Exploration
• Solutions: – Interactive Learning– Contextual Zooming– Gesture Recognition
• Conclusions and Future Work
JN 04/19/23
Knowledge Systems Lab
Interactive Learning
• Users can teach the computer to recognize abnormal brain tissue
• They can refine the learning systems results interactively
• They can save/load agents for background diagnosis on a database of medical images or to allow expert analysis in the absence of a well-paid expert
JN 04/19/23
Knowledge Systems Lab
Contextual Zoom
• They can zoom for precisely viewing and selecting targets/non-targets while maintaining context of the entire image
• The interface facilitates task performance through interactive and customizable display of segmentation results
• This system can be used with any 2D images and even with 3D datasets with some minor alterations
JN 04/19/23
Knowledge Systems Lab
Gesture Recognition
• Can use intuitive gesturing to interact with complex, 3D data
• Can interact by simply moving their hand in front of a camera, requiring no physical device manipulation
• Easily replicated and distributable
• Mapping gestures to interaction is an independent stage
JN 04/19/23
Knowledge Systems Lab
Gesture Recognition
• Dynamic Gesture Recognition
• Other interface applications include: graspable interfaces, 3D avatar / MoCap, multi-object manipulation in virtual environments, and augmented reality
JN 04/19/23
Knowledge Systems Lab
Platform
• Med-LIFE integration effort– Gesture Recognition has already been
integrated into Med-LIFE’s Exploration tab– Contextual Zoom and Interactive learning
have been combined, but not yet integrated into Med-LIFE’s learning tab
• Med-LIFE will function as a single application for medical image analysis