A multispectral image enhancement approach to visualize tissue structures

Pinky A. Bautista1, Tokiya Abe1, Yukako Yagi1, John Gilbertson1, Masahiro Yamaguchi2, and Nagaaki Ohyama2

1 Massachusetts General Hospital 2 Technology Institute of Technology

Multispectral Imaging (MSI)

Originally developed for space-based imaging

Multiple grey-level images are captured at different wavelengths

Allows extraction of additional information which the human eye fails to capture.

Filter sensitivities

3 grey-level images

N>3 grey-level images

RGB imaging Mutispectral imaging

MSI allows greater

flexibility for image analysis as compared to RGB imaging3 broadband

filtersN Narrowband filters

Objectives

To digitally enhance an H&E stained multispectral image such that collagen fiber can easily be differentiated from the rest of the eosin stained tissue components.

Show the capability of multispectral imaging to differentiate tissue structures with minute colorimetric difference.

Multispectral Microscope Imaging system

Olympus BX-62 optical microscope controlled by a PC

16 interference filters

2kx2k pixel CCD camera

*Used in the experiment

Enhancement Method yx,yx,yx,yx, t-tWtte

~

Reference: Masanori Mitsui, Yuri Murakami, Takashi Obi, et.al, “ Color Enhancement in Multispectral Image Using the Karhunen-Loeve Transform,” Optical Review Vol.12, no.2, pp.60-75, 2005

yx,t Original spectral transmittance at location x,y (16-band)

yx,et Enhanced version

estimated spectral transmittance using M (M<N) KL vectors derived from the transmittance data of the selected tissue components

Spectral residual error

W NxN weighting factor Matrix, i.e. N=16

tvrrt

M

1iii~

Controls the color of enhanced areas

Experiment

1. Training Phase 2. Testing Phase

Collection of 16-band transmittance spectra samples of the identified H&E stained tissue components

Derivation of the KL vectors

Identification of the appropriate number of KL vectors, i.e. M-KL vectors

Perform multispectral enhancement on 16-band images using the M-KL vectors derived in the training phase

Transform the multispectral enhanced image into its equivalent RGB format for visualization

Examine the spectral residual error characteristics of the

different tissue components

Derivation of KL vectors

KL vectors were derived from the transmittance of these tissue components

Transmittance spectra of the different tissue components

Training data

fiber

Subject for enhancement

Not Subject for enhancement

RGB format of the 16-band MS image of a

Heart tissue

NucleusCytoplasmRBCs, etc.

Tissue components transmittance spectra

0

0.2

0.4

0.6

0.8

1

1.2

1 3 5 7 9 11 13 15

Band number

Tra

nsm

ittan

ce v

alue Nucleus

Striated Muscle

Red blood cell (RBC)

White area

Collagen fiber

others

Each tissue component is represented with 200 transmittance spectra samples.

structures found in white areas

Spectral Residual Error

-0.03

-0.02

-0.01

0.00

0.01

0.02

0.03

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Band number

Ave

rag

e sp

ectr

al r

esid

ual

err

or

Nucleus

Striated muscle

Red blood cell (RBC)

White area

Collagen fiber

others

The spectral residual error for fiber peaks at band 8

structures found in white areas

Appropriate number of KL vectors was investigated……

5-KL vectors were found to produce distinct peaks on the spectral residual error of collagen fiber

Result (heart tissue)

H&E stained Digitally enhanced

Striated muscle and Collagen fiber which are both stained with Eosin in an H&E stained slide are impressed with different

shades of color when digitally enhanced

Collagen fiber

Striated muscle

Striated Muscle

Collagen fiber

2kx2k pixels 20x

magnification

Results

H&E stainedMT stained Digitally enhanced

Serial Section

referenceTissue areas highlighted in the digitally enhanced image correspond to areas

emphasized by the MT stain

Result (Magnified)

not clearly differentiated

differentiated differentiated

Original H&E stained

Enhanced H&E stained image

MT stained

Tissue structures with minute color difference is differentiated using Multispectral information

reference

RGB and Multispectral

Enhanced using RGB information

Enhanced using Multispectral

information

Original H&E

stained image

MT stained image

Serial Section

RGB and Multispectral Enhanced using RGB

informationEnhanced using

Multispectral information

Original H&E stained image

Not clearly differentiated Clearly differentiated

0

0.2

0.4

0.6

0.8

1

1.2

1 2 3 4 5 6 7 8 9 10 1112 1314 1516

Band number

Spe

ctra

l tra

nsm

ittan

ceStriated muscle

Collagen fiber1

Collagen fiber2

Spectral transmittance

There is a slight difference in the spectral configurations between the labeled fiber1 and fiber2 areas

Conclusion With multispectral imaging it is possible to differentiate tissue structures with minute colorimetric difference

The current enhancement scheme makes it possible to differentiate tissue structures that are less likely differentiated with RGB imaging

Future work

Work with more tissue images to validate the current result

Investigate further the meaning of spectral residual error in relation to tissue differentiation

Investigate possible application of the residual error configurations to select important bands to classify/segment specific tissue structures

We thank CAP foundation for making it possible for us to attend this conference.

THANK YOU….

Weighting matrix Variation

H&E stained Digitally enhanced

Color of target areas

can be varied by

manipulating the

weighting

matrix W

yx,yx,yx,yx, t-tWtte

~Spectral enhancement

Results

H&E stained MT stained Digitally enhanced

Serial Section

reference

Tissue areas highlighted in the digitally enhanced image correspond to areas emphasized by the MT stain

Result (kidney tissue)

Training data were extracted from another MS image of a kidney tissue; training and test images belong to the same slide

H&E stained Digitally enhanced

Result (kidney tissue)

H&E stained Digitally enhanced MT stained