University of Applied Sciences of Wiesbaden Department of Computer Science Image Processing

University of Applied Sciences of WiesbadenDepartment of Computer ScienceImage ProcessingDetlef Richter

New Applications of Digital Image Processingin Technology and Medicine

CERNIT Division

CH – 1211 Genève 23August 15th, 2003

Agenda-----------------------------------------------------------------------------------------1. Introduction2. Hardware Development3. Basic Algorithms4. Spectral Sensitivity5. Image Sensors6. Stereo Vision7. Examples of Applications

7.1 Automated Assembly7.2 Sound Track Restoration7.3 Computer Based Learning7.4 Image Processing of medical Images7.5 medical Navigation

8. Summary

1. Introduction ----------------------------------------------------------------------------------------1975 Systematic Scientific Development of Image Processing

Problem : Availability of sufficient fast ADC / DAC Hardware

1980 Introduction of Image Processing inIndustrial Assembly Lines, Production Control, Quality Reliability

1985 medical Image Analysis

For Example : Production of Middle Class Car

1985 West Europe 65 h / Japan 35 h

2002 General Motors, Eisenach 8 h

by automation, part of which is done by robotics and image processing

2. Hardware Development (1)

• Dedicated Image Processing Unit

• Image Display on Video Monitor

• Image Data Transfer to Computer via DMA

• Image Processing Unit Control via C-Bus Extension

• Bottle Necks : Transfer Rates of DMA and Computer Bus System

ADC DACMEM.

DMA

Interf.Contr.

Interf.

DMA

Interf.

Contr.

Interf.

Bus-System

MEM. CPU Graphic

Board

Imaging

SensorVideo

Monitor

Computer

Monitor

Hardw01.DS4

Process

Control

Imaging

Unit


• Frame Grabber Boards

• Occasional Local CPU on Board

• Improvement of Fast Processing

• Disadvantage of Programming

• Decreasing use of Video Output

• No Separate Video Monitor Necessary

• Image Output and Programming in Different Windows on Same Screen

• Bottle Neck : Computer Bus System if no Local CPU on Board

ADC MEM.Contr.

Interf.

Bus-System

MEM. CPUGraphic

Board

Imaging

Sensor

Video

Monitor

Computer

Monitor

Hardw02.DS4

Process

Control

DAC

local

CPU

Frame

Grabber

Board


• No Memory on Board

• Image Data and Programs in Same Memory

• Fast Data Transfer via Computer Bus,

PCI-Bus : 1.3 MByte, Transfer Rate Sufficient for Transferring 3 Video Frames at Same Time,

e. g. RGB-Images

ADCContr.

Interf.

Bus-System

MEM. CPU Graphic

Board

Imaging

Sensor

Computer

Monitor

Hardw03.DS4

Process

Control

Frame

Grabber

Board

3. Basic Algorithms ( 1 )

Modeling of a Scene( A-Priori-Knowledge )

Global Preprocessing

( Noise Reduction, Edge Filtering, Image Transforms etc. )

Segmentation

( Objects / Background )ROI or VOI

Extraction of Attributes / Values

Analysis of Attributes / Values

( Classification )

Interpretation, Action

••

BinarisationLow Pass Filtering ( Noise Reduction )

2D High Pass Filtering ( Edge Detection )3D High Pass Filtering ( Shape Detection )

Hough TransformFourier Transform

Autocorrelation FunctionSpecific Procedures

••


• Video Image

• Resolution 786 x 576 Pixels

• 8 Bit Gray Level Resolution


• Global Filtering

e. g. 2D-Low Pass Filter ( Median )

• Scan the Image with n*n-Matrix

• Apply Matrix for Each Pixel

• Generate New Image Frame

( Noise Reduction )


• Global Filtering

e. g. 2D-High Pass Filter

• Scan the Image with n*n-Matrix

• Apply Matrix for Each Pixel

• Generate New Image Frame

( Edge Detection )


• Global Transformation

e. g. Hough Transform

for Lines and Circles

• Transform Each Pixel of an Edge

into a Hough Matrix

• Analyze the Hough Matrix

• Gain Mathematical Equations

of Edges


• Calculate Vanishing Points of Lines

• Define optimal Cut-Out of Source Image

• Define Rectangular Area in Target Image

• Interpolate all Pixels in Target Image by Resampling


• Global Transformation

e. g. Fourier- Transform

( 1D and 2D )

• FFT

• Transform Each Line and/or Row

from Geometric Domain

into Frequency Domain

• Analyze Frequency Domain

• Extract Frequency Attributes

or Change Frequency Attributes

and Execute Inverse Transform

• Gray Levels with logarithmic enhanced Representation

4. Spectral Sensitivity ( 1 )

• Visible Electromagnetic Wavelengths 400 nm ( violet-blue ) to 750 nm ( red )

Monochromatic Cameras

• Gray Level Video Camera (Luminance Signal Output According Integral Sensitivity )

• IR-sensitive Cameras for medical Applications ( One Channel, Luminance or False Color Output )

• Video Cameras with IR-Long Pass Filter for Technical Applications ( One Channel, Small Bandwidth )

Spectral Sensitive Cameras

• One Chip Cameras with R,G,B-Filter Mask, FBAS-Output ( Three Channels coded on one Line )

• Three Chip Cameras, R,G,B-Output ( Three Channels )

• Landsat Images, IR-Channels ( up to Seven Channels )

4. Spectral Sensitivity ( 2 )

CCD-Chip

R R R RG G G GB B B B

Reduced Resolution

Slit Mask

Orig-inalReso-lution

Slit Mask Camera

CCD-Chip

CCD-Chip

CCD-Chip

R-Filter

G-Filter

B-Filter

Objektiv

Bild620a.DS4

Beam Splitter

5. Image Sensors

• Video Cameras ( 2D )– Reflection of Incident Light on Surface– Monochrome Cameras ( Gray Level, Modified IR Level )– Color Cameras ( One Line FBAS, Three Lines R,G,B )– Infrared Cameras ( monochrome )

• Line Scan Camera ( 1D )

• Roentgen Sensitive Image– 2D Transparent Shadow of 3D Object– Large Area Solid Roentgen-Sensitive Devices

• Computer Tomography ( CT ), since 1972– 2D Slices of 3D Object, Volume Image, X-Ray Scattering by High Z-Nuclei

• Magnetic Resonance Tomography ( MRT ), since 1980– 2D Slices of 3D Object, Volume Image, “Light Emitting Volume by Hydrogen Nuclei”

• Positron Emission Tomography ( PET ), since 1978– 2D Slices of 3D Object, Distribution of radionuclides, shows chemical activity,

“Functional Image”

• Ultra Sound ( 2D ), since 1960– Reflection on Tissue Discontinuities, Run Time Measurement of Sound

5.1 Large Area Solid Roentgen-Sensitive Devices

• Large scale Roentgen sensitive sensor

• Sensor elements 200 μ * 200 μ

• Resolution 2 K *2 K pixel

• Grey level resolution 16 Bit

• Data 16 MB per Image

• Roentgen energy 80 KeV → 400 KeV

• approx. 3 frames per s ( 1K*1K 7 frames )

• Quality assurance / Production Control / Materials Research and Testing

• Medical application

5.2 PET Images

• 18F-Fluorodesoxyglucose

• Decay

• p → n + e+ + ν

• e+ + e- → 2 * 511 KeV

• Ekin(e+) = 0,633 MeV

• mean free path in H2O : 2,4 mm

• τ½ = 109,7 min.

• search for diseases before symptoms appear ( e.g. Alzheimer disease, micro metastases etc. )

• Problems :Mathematical modeling of the physiological procedure

OF 10n 8p18Z

9n 9p18Z

5.3 US Images

• US-Picture Abdomen

• High Signal to Noise Ratio

• Frequency of Sound 8 to 13 MHz

• Power 10 to 50 mW

• Velocity of Waves Dependent on Tissue

• Measurement of Time with supposed constant Velocity

• Ratio of Emitting to Receiving Time 0.1%

6. Stereo Vision ( 1 )

• Model of Pinhole Camera with Radial Symmetric Distortion of the lenses

• Calibration of Intrinsic Mathematical and Physical Parameters :

– Width to Height Ratio of Pixels– Intersection of Optical Axes of the

Lenses with the Surface of the Sensor Chip

– Image width of the lenses– Coefficients defining Radial Symmetric

Distortion– Position and Orientation of the

Cameras

6. Stereo Vision ( 2 )

• Use chessboard like pattern of known size

• Apply Hough Transform

• Find the known number of edges of known direction

• Calculate all intersections

• Find subpixel precise all corners

• Calculate position and orientation of the camera

7.1 Automated Industrial Assembly ( 1 )

• Top View of an Industrial Precise Mechanical Production Line

• Problem of Perspective Distortion : Recognition of exact Positions in x,y-Plane

• Problem of Positioning the Robot Hand for Automatic Production and Control within 3D Space

• Parts Recognition, Identification and Measurement using Bottom Illumination by IR-Light

7.1 Automated Industrial Assembly ( 2 )

• Calibration of Monocular or Binocular Vision System (Position, Orientation, Focus Length of Camera ) with Respect to the Coordinates of Assembling System

• Calibration of Robot Coordinate System with Respect to Vision System

• Development of Customer-Specific Recognition Algorithms

• Optimizing of Lighting

• Test of Complete System

7.2 Sound Track Restoration ( 1 )

tonsp03e.DS4

Width of sound track 2.5 mm

Width of frame22 mm

Width of carrier 35 mm

Frameheight16 mm

Frameperiodicity

19 mm

Projection speed 24 frames / s. = 45.6 cm/s

Sensor

AmplifierLamp withslit aperture

EIZO

Video-Monitor

CPU

Frame-grabber Disc Video Sound

8 Bit CCDLine-Camera

512 PixelsAudio

Control

PC

VisualControl

tonsp04e.DS4

Film Scanner


• Reproduction Speed 24 Frames / s

• Sound Track Scanning with Line Camera 6 Frames / s

• Scanning of Track Width 512 Pixels per Line

• Gray Level Resolution 8 Bit / Pixel = 256 Gray Levels

• Nyquist Frequency 24 KHz

• Analog Cut off Frequency 15 KHz

• Frequency Resolution 2000 Lines per Frame

• Data Transfer Rate ( Camera – Disk Memory ) 6 MByte / s

• Required Storage Capacity 24 MByte / 1 Second of Sound


Left : Intensity Code Sound Track of the Speech of Albert Einstein 1930 on behalf of the Opening Ceremony of the Broadcasting Fair in Berlin ( Scratches, Spots, Fibers )

Right : Restored Sound Track within ROI, Conserving the Authenticity of First RecordingConverting the Sound Track in Digital Audio Data


Left : Twofold Double Sided Variable Area Code with Faults on Sound TrackRight : Multifold Double Sided Variable Area Code with Faults on Sound Track

7.3 Computer Based Learning ( 1 )

• Analysis of Time Dependent Color Change caused by Chemical Reactions

• Color Sensitive Chemical Indicator

• Time Dependent Addition of Reagents

• Analysis of Colors in Defined Color Space( R, G, B )

7.3 Computer Based Learning ( 2 )

• Example : Reagent Phenolphthalein, Measurement Time 300 s, Sample Intervalls 500 ms

• Create Colored Animation of Chemical Reaction

7.4 Image Processing of medical Images (1)

Methods of Segmentation :

• Body Surface : Change of Gray Level Distribution

• Vessels, Tracer of Vessels : Tube Model with slowly varying Diameter and Ramifications

Filament Model, Transition between both Models possible

- Seed Algorithm with Use of Contrasting Injection

- 3D rays in forward Direction

- Texture Analysis on Surface of Spheres

7.4 Image Processing of medical Images (2)

• Optic nerve : 3D interactively directed rays, no defined contrast

• Ventricles : 3D-growing region, Problem : Running out in Filament Structures

• Tumors : Slowly and unnoticed growing, backing down in liquor holes ( ventricles ), first complaints if burdening nerves ( optic nerves, auditory nerves )

Threshold based and edge detection based Procedures are without success.

Manual contouring is time consuming but a standard procedure.

New : Getting interactive significant numerical attrbutes for texture analysis, 2D-texture analysis, 2D reconstruction of Tumor

Disadvantage : No proper Contours detectable, manual completion necessary

• Liver : Definition of liver segments by blood vessels ( maximum 5 out of 7 liver-segments are resectable )

7.5 medical Navigation ( 1 )

• Besides infrared guided systems exist other navigational systems :

• Mechanical, Stereotactical ( direct contact )• Electromagnetical ( electromagnetic disturbance )• Laser Guided ( laser beam )

7.5 medical Navigation ( 2 ), Components

• IR-based stereo vision system with two CCD cameras with synchronised H-Sync-Signals

• four camera system under development

• conventional calibration of cameras, i. e.– spatial position and orientation– focal length – radial symmetric distortion of the optical lenses

7.5 medical Navigation ( 3 ), Tracker

• Tracker for biopsy needles with IR-LEDs, adapter for robot based guidance, force / torque sensor for feed back

• ( min. 3 LEDs, max. 6 LEDs )

• Wavelength of 895 ± 45 nm

• Longpass filter with cut-off frequency of 830 nm

• CT-volume image data( DICOM III Standard )

• Registration of patients position

• Visualization of navigation

7.5 medical Navigation (4 ), Aim of Brachytherapy

• Irradiation of recurrent or inoperable tumours,

• preserving normal tissue ( cells ),

• avoiding pre-irradiated tissue,

• using biopsy needles with inner diameter of 2.0 mm and outer diameter of 2.1 mm.

using np z== =11577 192Ir as radioactive source

t ( Ir ) = 74,2 da ys1/2

192

External therapy Brachytherapy

Radiation energy 18 MeV 0.6 MeV

Daily applications 1 2

Duration of one application 3 directions, each 10 - 20 s 5 - 15 min

Dose of one appl. 1,8 Gy 4 - 7 Gy

Duration of application 5 - 6 weeks 1 week

Total Dose in Gy 45 Gy 30 - 40 Gy

4. Mathematical Procedure -----------------------------------------------------------------------------------------------------------------------------

1. Step :

Calibrate the stereovision system

2. Step :

Calibrate the tracker,getting a precise tracker model T, i. e. calculate the 3D-coordinates of the LEDs of the trackernoted by : T = { Ti | Ti R3, i = 1, ..., n }

Repeat this step 50 times for getting a higher precision in the coordinates

Use a-priori-knowledge about the model for matching corresponding points

3. Step :

Calibrate the equation of the biopsy needle ( assumed to be a linear one ),with respect to the tracker modelwith different lengths of the needle

4. Step : ( Application )

Find the 3D coordinates of actual tracker position PP = { Pi | Pi R3, i = 1, ..., n }

Superimpose the actual tracker position P to the model T with optimising

getting the precise position and orientation of the tracker in 3D

Calculate the precise position of the end of the biopsy needle and its orientation

MinimumP - T2

ii

7.5 medical Navigation ( 5 ), Evaluation

High precision of position and orientation measurement

7.5 medical Navigation ( 6 ), Visualization ---------------------------------------------------------------------------------------------------------------------------------

--Definition of a CT tomogram data set intersecting plane containing the biopsy channel

Resolution of CT tomogram data set is not homogeneoustime consuming 3D interpolation would be necessary

Homogenisation of the data set ( isotropic voxels ), increasing the data by a factor 5 to 10 and using the nearest voxel of intersection plane without interpolation

7.5 medical Navigation ( 7 ), Example of Visualization

7.5 medical Navigation ( 8 ), Projected Lay-out

7.5 Medical Navigation ( 9 ), Registration of Patient

• 3D-Segmentation of landmarks ( containing IR-LEDs ) within the CT-volume-data of the interesting region of the body, defining the position of the data-cube and of the tumor within the body.

• Segmentation of the landmarks by the stereo vision system during medical treatment, defining the absolute position of the tumor within the CT-data cube.

• Transfer Navigation Data from Vision System into CT Volume Date by

xCT = R * xCam + T

• Visualize Navigation of Biopsy Needle within CT-volume Data

7.5 medical Navigation ( 10 )

• 8. Procedure of Virtual Navigation and Further Steps ---------------------------------------------------------------------------------------------------------------------------------

• Interactive segmentation of tumor and other risk structures by the medical specialist.

• Automatic segmentation of bones.

• Calculation of possible 3D access paths to the tumor.

Iteration of verification of the position of the biopsy needle within the tumour by CT duringapplication.

Automatic positioning of the biopsy needle by a robot according to the known access paths formanual interactive application by physician ( projected ).

Automatic positioning of the biopsy needle by a robot according according to the known accesspaths using online force feed back ( projected ).

8. Summary ( 1 )

• Hardware Development

• Basic Algorithms

• Spectral Sensitivity

• Image Sensors

• Stereo Vision

• Examples of applications

8. Summary ( 2 )

• Fast Developing Discipline

• Many Known Algorithms Exist for Technical Applications ( mainly without any interaction )

• Brand New Applications have to be developed for medical Diagnosis and Therapy

• Only very few standard Algorithms Exist for medical Applications ( interactions necessary )

• Creativity, Intuition and Experience is necessary to Solve Problems

• Many Attainments of other Disciplines ( Electrical Engineering, Math, Physics, Chemistry ) are required

University of Applied Sciences of Wiesbaden Department of Computer Science Image Processing

Documents

Transcript of University of Applied Sciences of Wiesbaden Department of Computer Science Image Processing