2065-36 Advanced Training Course on FPGA Design and VHDL...

2065-36

Advanced Training Course on FPGA Design and VHDL for HardwareSimulation and Synthesis

Fabio Mammano

26 October - 20 November, 2009

University of Padua Faculty of Medicine and Surgery Department of Physics via Marzolo 8 35131 Padova Italy

Three-dimensional deconvolution in FPGA

ThreeThree--dimensional dimensional deconvolutiondeconvolution

in FPGAin FPGA

Deconvolution 2

Summary of deconvolutionSummary of deconvolutionDeconvolutionDeconvolution isis a a computationallycomputationallyintensive intensive imageimage processing processing techniquetechnique thatthatisis beingbeing increasinglyincreasingly utilizedutilized forfor improvingimprovingthe the contrastcontrast and and resolutionresolution of of digitaldigitalimagesimages capturedcaptured in the microscope. The in the microscope. The foundationsfoundations are are basedbased uponupon a suite of a suite of methodsmethods thatthat are are designeddesigned toto removeremove or or reverse the reverse the blurringblurring presentpresent in microscope in microscope imagesimages inducedinduced byby diffractiondiffraction..

The basic concepts surrounding The basic concepts surrounding acquisition of serial optical sections for acquisition of serial optical sections for deconvolution deconvolution analysisanalysis are are presentedpresentedwithwith a a schematicschematic diagramdiagram in Figure 1. in Figure 1.

•• The specimen The specimen isis anan idealizedidealized cellcell fromfromwhichwhich a a seriesseries of of opticaloptical sectionssections are are recordedrecorded alongalong the the zz--axisaxis of of anan opticalopticalmicroscope. microscope.

•• ForFor eacheach focalfocal planeplane in the specimen, in the specimen, a a correspondingcorresponding imageimage planeplane isisrecordedrecorded byby the detector and the detector and subsequentlysubsequently storedstored in a data in a data analysisanalysiscomputer. computer.

•• DuringDuring deconvolutiondeconvolution analysisanalysis, the , the entireentire seriesseries of of opticaloptical sectionssections isisanalyzedanalyzed toto create a create a threethree--dimensionaldimensional montagemontage..

Deconvolution 3

Modeling the effect of an optical system: 3D inputModeling the effect of an optical system: 3D input--output relationship output relationship

The 3D convolution operation is defined as:The 3D convolution operation is defined as:

� � � � � �zyxszyxizyxo ,,,,,, ��

� � � � � �, ,

, , , , , ,u v w

o x y z i u v w s x u y v z w du dv dw� � � ��

� �, , :i x y z

� �, , :s x y z

� �, , :o x y z

InputInput light distributionlight distribution

Impulse responseImpulse response (3D PSF): describes a process causing (3D PSF): describes a process causing the output distribution to be different from the original the output distribution to be different from the original scene. scene.

OutputOutput light distributionlight distribution

Model equation:Model equation:

The impulse response (3D PSF), is the output light distribution The impulse response (3D PSF), is the output light distribution in image space for a in image space for a point source input in object space.point source input in object space.

Deconvolution 4

ZZ--stack decompositionstack decomposition

� � � � � �zyxszyxizyxo ,,,,,, ��

� � � � � � � � � � � � � �0 0 1 1 1 1, , , , , , , ...zo x y i x y s x y i x y s x y i x y s x y� � � ��

We can start by trying to decompose the 3D model equationWe can start by trying to decompose the 3D model equation

into a set of throughinto a set of through--focus twofocus two--dimensional series expansions (zdimensional series expansions (z--stack)stack)

Deconvolution 5� � � � � � � � � � � � � �0 0 1 1 1 1, , , , , , , ...zo x y i x y s x y i x y s x y i x y s x y� � � ��

� �0 ,s x y � �1 ,s x y� � �2 ,s x y�

� �1 ,s x y� �2 ,s x y� �3 ,s x y

� �3 ,s x y�

Function sk(x,y) , k = �N, �N+1…, 0, 1, 2 …N, are sections through the 3-D PSF taken in planes orthogonal to the optic axis of the system

optic axis

Deconvolution 6

To convert these equations into an algorithm that can be used inTo convert these equations into an algorithm that can be used in practice, we will practice, we will make two simplifying assumptions:make two simplifying assumptions:

1 1i o� �

1 1i o� �

2.2. Light originating from planes immediately above or below the plaLight originating from planes immediately above or below the plane of ne of focus can be approximated by images taken while focusing on thesfocus can be approximated by images taken while focusing on these planes, e planes, i.e.i.e.

1.1. OutOut--ofof--focus light contribution other than those from the focus light contribution other than those from the adjacent planesadjacent planesare negligible, i.e. terms with index different from 0 or are negligible, i.e. terms with index different from 0 or 1 can be ignored:1 can be ignored:

� � � � � ��

0 0

1 1

1 1

, , ,

, ,

, ,

zo x y i x y s x y

i x y s x y

i x y s x y� �

� �

� �

� � �

� �

((CastelmanCastelman, 1979), 1979)

(this assumption is not strictly necessary and will be relaxed l(this assumption is not strictly necessary and will be relaxed later).ater).

Deconvolution 7

z=01o�

0o

1o�

ZZ--stack decompositionstack decomposition

� � � �0 0 0 1 1 1 1F i s F o o s o s� � � ��

we write the simplified equationwe write the simplified equation

0 0 0 1 1 1 1o i s o s o s� � � ��

Now we take the Fourier transform, Now we take the Fourier transform, FF, of both sides:, of both sides:

0 0 0 1 1 1 1i s o o s o s� � � ��

in the formin the form

Taking plane Taking plane zz=0=0 as the reference focal plane,as the reference focal plane,

Deconvolution 8

� � � � � �0 0 0 1 1 1 1F i F s F o o s o s� � � ��

� � � � � �0 0 1 1 1 10

1F i F o o s o sF s� � � ��

� � � �1

0 0 1 1 1 10

1i F F o o s o sF s

��

� � � � � � ��

� �

By applying the By applying the convolution theoremconvolution theorem we obtainwe obtain

Taking the Taking the inverse Fourier transforminverse Fourier transform, , FF ��11, of both sides finally yields:, of both sides finally yields:

and thereforeand therefore

Deconvolution 9

With the stated assumptions, the inWith the stated assumptions, the in--focus light distributionfocus light distribution iioo can be recovered can be recovered (restored) from the equation:(restored) from the equation:

In practiceIn practice, nearest neighbor deconvolution is performed using a modified f, nearest neighbor deconvolution is performed using a modified form of orm of the above equation: the above equation:

where where cc--11 , , cc+1+1 and and cc00 are empirical factors and all are empirical factors and all PSFsPSFs are normalized. are normalized.

� � � ��

*01

0 0 1 1 1 1 1 1 20 0

F si o c o s c o s F

F s c�

� � � � � �

� ��

� � � �1

0 0 1 1 1 10

1i o o s o s FF s

��

� ��

� �

CoefficientsCoefficients cc11 are used to limit errors caused by overare used to limit errors caused by over--compensating outcompensating out--ofof--focus focus contributions and their value falls generally in the range from contributions and their value falls generally in the range from 0.010.01 to to 0.10.1..

Nearest neighbor (NN) Nearest neighbor (NN) deconvolutiondeconvolution

Deconvolution 10

usually complex 0.045�

cc00 is required to handle exceptions caused by the possible presencis required to handle exceptions caused by the possible presence of zeros ine of zeros inFF((ss00)); values for ; values for cc00 fall in the range from fall in the range from 0.0450.045 to to 0.0500.050. Note that:. Note that:

•• Too large a value for Too large a value for cc00 causes loss of details and yields blurry images.causes loss of details and yields blurry images.

•• Too small a value causes amplification of random noise into brigToo small a value causes amplification of random noise into bright spots, ht spots, rings, or patches.rings, or patches.

� ��

*01

20 0

F sF

F s c�

� ��

Deconvolution 11

MultiMulti--neighbour neighbour deconvolutiondeconvolution

In summary, this procedure requires:In summary, this procedure requires:

•• in focus and outin focus and out--ofof--focus image collectionfocus image collection

•• in focus and outin focus and out--ofof--focus PSF measurementfocus PSF measurement

� � � ��

*01

0 0 21 0 0

N

j j j j j jj


F s c�

� � � � � ��

� ��

Relaxing condition 1 and generalizing to Relaxing condition 1 and generalizing to NN neighbours yields the result:neighbours yields the result:

Deconvolution 12

Serial focal sequence of fluorescence images of Serial focal sequence of fluorescence images of a 200 nm bead used to measure a 200 nm bead used to measure xx--yy sections of sections of the 3D PSF at various distances above and the 3D PSF at various distances above and below the focal planebelow the focal plane

ThroughThrough--focus behaviour of point imagesfocus behaviour of point images

MeridionalMeridional ((yy--zz) section through the PSF. This ) section through the PSF. This pseudopseudo--color image was generated color image was generated electronically from the series of bead images electronically from the series of bead images shown at left. shown at left.

The asymmetry in the axial direction is due to spherical aberratThe asymmetry in the axial direction is due to spherical aberration and is very common ion and is very common in imaging biological specimens. There is additional path lengthin imaging biological specimens. There is additional path length through a layer of through a layer of water, glycerol or other mounting medium, which leads to increaswater, glycerol or other mounting medium, which leads to increased spherical ed spherical aberration.aberration.

Deconvolution 13

•• FindingFinding the (closest approximation to) the focal plane; by definition, the (closest approximation to) the focal plane; by definition, this is the this is the plane where the lateral extent of the PSF is minimal and its peaplane where the lateral extent of the PSF is minimal and its peak value is k value is maximalmaximal

The 3D PSF must be measured on the same system used to acquire tThe 3D PSF must be measured on the same system used to acquire the images to be he images to be deconvolved. For fluorescence imaging, this process requires:deconvolved. For fluorescence imaging, this process requires:

Measuring the 3D PSFMeasuring the 3D PSF

•• Setting the stepSetting the step of the of the zz--stack: stack: NyquistNyquist theorem.theorem.

•• EmbeddingEmbedding subsub--resolution fluorescent beads into the medium of interest. If thiresolution fluorescent beads into the medium of interest. If this s is cytoplasm, the beads can be incorporated by cell is cytoplasm, the beads can be incorporated by cell electroporationelectroporation..

•• SymmetrizationSymmetrization: in the absence of aberrations, the PSF has rotational symmetry: in the absence of aberrations, the PSF has rotational symmetryaround the optical axis; raw around the optical axis; raw xx--yy PSF sections are made symmetric by a circular PSF sections are made symmetric by a circular averaging procedure.averaging procedure.

•• NormalizationNormalization: in an ideal widefield microscope, the total integrated intensi: in an ideal widefield microscope, the total integrated intensity at ty at each outeach out--ofof--focus plane is constant.focus plane is constant.

Deconvolution 14

A typical 11A typical 11××11 kernel11 kernel

Deconvolution 15

Deconvolved(4 neighbors)

ConfocalRaw images10 μm

Foca

l pla

ne-2

.5 μ

m-5

μm

+2 μ

m+4

μm

Algorithm validation: optical sections of a 15 Algorithm validation: optical sections of a 15 ��m m test spheretest sphere

Deconvolution 16

Application to biological specimensApplication to biological specimens

Conventional (wide-field) fluorescence image of Convallaria Rhizom (left), deconvolved with the nearest neighbour algorithm (right).

ZZ--stack of mitochondria stack of mitochondria targeted with targeted with cameleoncameleon

10 μm

20 μm

Deconvolution 17

NN NN deconvolutiondeconvolution in real time: FPGAin real time: FPGA

CCD camera target frame rate: CCD camera target frame rate: 60 frames per second (fps), to obtain 30 fps of NN 60 frames per second (fps), to obtain 30 fps of NN deconvolveddeconvolved imagesimages

CCD size: CCD size: 512512�� 512 pixels (512 pixels (binnigbinnig = 2= 2��2 on a 2 on a MpixelMpixel sensor)sensor)

Reasonable kernel size: Reasonable kernel size: 1111�� 11 pixels11 pixels

Number of multiplications (and additions) per frame: Number of multiplications (and additions) per frame: (512(512�� 512512) �� (1111�� 11)11) = 31.7 M= 31.7 M

Computational parallelism is a must!Computational parallelism is a must!

At 60 fps:At 60 fps:31.7 M31.7 M �� 60/s = 1.9 G/s60/s = 1.9 G/s

One deconvolved image requires 3 consecutive images. However, thOne deconvolved image requires 3 consecutive images. However, the last one of a e last one of a frame triplet is also the first one of the following triplet. Thframe triplet is also the first one of the following triplet. Therefore deconvolved erefore deconvolved frames are generate at half the acquisition frame rate.frames are generate at half the acquisition frame rate.

Deconvolution 18

z=01o�

0o

1o�

Breaking down the problem into time slotsBreaking down the problem into time slots

constant matrixconstant scalar

Frame exposure intervals

1o�

0o1o�

focal plane

� � � ��

*01

0 0 1 1 1 1 1 1 20 0


F s c�

� � � � � �

� ��

Deconvolution 19

Overview of Slots 1, 2 and 3Overview of Slots 1, 2 and 3

focal plane

•• Slot 1: collect light to form image Slot 1: collect light to form image oo��11

•• Slot 2: download image Slot 2: download image oo--11 data while collecting light to from image data while collecting light to from image oo00

•• Slot 3: download image Slot 3: download image oo00 datadata while collecting light to form image while collecting light to form image oo+1+1•• ………………..

out out out

Light collectionImage ready

1o�

0o1o�

Deconvolution 20

Slot 2 in detailSlot 2 in detail

1 1 1 1i c o s� � � ��

•• Collect light to form image Collect light to form image oo00;;•• in parallel, as in parallel, as oo––11 data come through, computedata come through, compute

1 1c o� �1s�

1o�

•• store the result into a memory buffer.store the result into a memory buffer.

To memorybuffer

� ��

1

*01

0 0 1 1 1 1 1 1 20 0i


F s c�

��

� � � ��

� ��

Deconvolution 21


1i �

0 1o i��

•• collect light to form image collect light to form image oo+1+1 ;;•• in parallel, as in parallel, as oo00 data come through, subtract the result (data come through, subtract the result (ii––11) obtained in ) obtained in

Slot 2;Slot 2;•• store also this new result into a memory buffer.store also this new result into a memory buffer.

1s�

� ��

1

*01

0 0 1 1 1 1 1 1 20 0i


F s c�

��

� � � ��

� �� Computed

in Slot 2

From memorybuffer

To memorybuffer

Deconvolution 22


1 1c o� �1s�

0 1o i��

0 1 1o i i� ��

0 1.o i��

� ��

1

*01

0 1 1 1 1 1 1 20 0i


F s c�

��

� � � ��

� ��

1 1 1 1i c o s� � � ��

•• subtract this new result subtract this new result from the partial difference (from the partial difference (oo00 –– ii––11) obtained in Slot 3.) obtained in Slot 3.

•• collect light to form a new image collect light to form a new image oo00’’ ;;•• as as oo+1+1 data from Slot 3 come through compute:data from Slot 3 come through compute:

From memorybuffer

0o1o�

1o�

0'o

Deconvolution 23

Slots 5 in detailSlots 5 in detail

•• the results is the the results is the first deconvolved imagefirst deconvolved image, which is sent to the host PC, which is sent to the host PC

•• collect light to form a new image collect light to form a new image oo--11’’

•• in parallel, convolve the in parallel, convolve the result of Slot 4, that isresult of Slot 4, that is

with matrix with matrix

(which must have been previously stored in memory)(which must have been previously stored in memory)

0 1 1o i i� ��

� ��

*01

20 0

F sF

F s c�

� ��

� � � ��

*01

0 1 1 1 1 1 1 20 0


F s c�

� � � � � �

� ��

��Computed in Slot 4

0o1o�

1o�

0'o1'o�

Deconvolution 24

FPGA example: convolution with a separable kernelFPGA example: convolution with a separable kernel

1 2( , ) ( ) ( )s j k s j s k�

2 1

2 11 1

( , ) ( ) ( , ) ( )N N

k jo m n s n k i j k s m j

� �

� � ��

Deconvolution 25

Vertical filterVertical filter

This 1D filter has kernel This 1D filter has kernel ss11 and multiplies all pixels in a column of and multiplies all pixels in a column of 11 pixels11 pixels by the by the kernel coefficients, followed by the adder tree that sums up thekernel coefficients, followed by the adder tree that sums up the results two by two.results two by two.

Deconvolution 26

Horizontal filterHorizontal filter

This 1D filter has kernel This 1D filter has kernel ss22 and its inputs are delayed and organized in rows through a and its inputs are delayed and organized in rows through a series of cascaded registers that reconstruct the spatial dependseries of cascaded registers that reconstruct the spatial dependence.ence.

Deconvolution 27

BooksPrinciples of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light, 7th edition., Born, M. and Wolf, E., Cambridge University Press, Cambridge, United Kingdom, 952 pages (1999). Video Microscopy (Methods in Cell Biology, 56)., Sluder, G. and Wolf, D. (eds), Academic Press, New York, 334 pages(1998).Deconvolution of Images and Spectra, 2nd edition., Jansson, P. (ed), Academic Press, San Diego, California, 514 pages(1997). Introduction to Fourier Optics., Goodman, J., McGraw-Hill, New York, 441 pages (1996).Digital Image Processing: Principles and Applications., Baxes, G., John Wiley and Sons, New York, 452 pages (1994).Digital Image Processing., Castleman, K., Prentice-Hall, Englewood Cliffs, New Jersey, 667 pages (1979).Review ArticlesImage-restoration methods: basics and algorithms., Boccacci, P. and Bertero, M., in Confocal and Two-Photon Microscopy: Foundations, Applications, and Advances, Diaspro, A. (ed), Wiley-Liss, New York, pages 253-269 (2002).A workingperson's guide to deconvolution in light microscopy., Wallace, W., Schaefer, L., and Swedlow, J., in BioTechniques, 31(5), pages 1076-1097 (2001).Contrast, resolution, bleaching and statistics in confocal microscopy., Pawley, J., in Focus on Multidimensional Microscopy, Cheng, P., Hwang, P., Wu, J., Wang, G., and Kim, H. (eds), World Scientific, Singapore, pages 134-141 (1999).Three-dimensional imaging by deconvolution microscopy., McNally, J., Karpova, T., Cooper, J., and Conchello, J., in Methods, 19, pages 373-385 (1999).Computational deblurring of fluorescence microscope images., Shaw, P., in Cell Biology: A Handbook, 2nd edition, Academic Press, San Diego, California, pages 206-217 (1998).Confocal microscopy and deconvolution techniques., Spector, D., Goldman, R., and Leinwand, L., in Cells: A Laboratory Manual, Volume 2, Light Microscopy and Cell Structure, Cold Spring Harbor Laboratory Press, Cold Spring Harbor, New York, pages 96.1-96.23 (1998).High resolution 3-D imaging of living cells by image restoration., Carrington, W., Fogarty, K., Lifshitz, L., and Tuft, R., in Imaging Living Cells, Rizzuto, R. and Fasolato, C. (eds), Springer-Verlag, Heidelberg, Germany, pages 30-50 (1998).

SelectedSelected referencesreferences

2065-36 Advanced Training Course on FPGA Design and VHDL...

Documents

Transcript of 2065-36 Advanced Training Course on FPGA Design and VHDL...