F usion of contrast-enhanced breast MR and mammographic ...kmarias/4_CPB_MEDIA.pdf · Christian P....

Medical Image Analysis 7 (2003) 311–340www.elsevier.com/ locate/media

F usion of contrast-enhanced breast MR and mammographic imagingdata

a,b , a,b a a*Christian P. Behrenbruch , Kostas Marias , Paul A. Armitage , Margaret Yam ,c d e aNiall Moore , Ruth E. English , Jane Clarke , Michael Brady

aMedical Vision Laboratory (Robotics), Engineering Science, Oxford University, Parks Road, Oxford OX1 3PJ, UKbDepartment of Surgery, Royal Free and University College Medical School, UCL, London NW3 2QG, UK

cMagnetic Resonance Imaging Centre, John Radcliffe Hospital, Headley Way, Oxford OX3 9DU, UKdBreast Care Unit, Churchill Hospital, Oxford OX3 7LJ, UK

eDepartment of Surgery, Churchill Hospital, Oxford OX3 7LJ, UK

Received 5 February 2001; received in revised form 25 February 2002; accepted 29 October 2002

Abstract

Increasing use is being made of Gd-DTPA contrast-enhanced magnetic resonance imaging for breast cancer assessment since it provides3D functional information via pharmacokinetic interaction between contrast agent and tumour vascularity, and because it is applicable towomen of all ages as well as patients with post-operative scarring. Contrast-enhanced MRI (CE-MRI) is complementary to conventionalX-ray mammography, since it is a relatively low-resolution functional counterpart of a comparatively high-resolution 2D structuralrepresentation. However, despite the additional information provided by MRI, mammography is still an extremely important diagnosticimaging modality, particularly for several common conditions such as ductal carcinoma in situ (DCIS) where it has been shown that thereis a strong correlation between microcalcification clusters and malignancy. Pathological indicators such as calcifications and finespiculations are not visible in CE-MRI and therefore there is clinical and diagnostic value in fusing the high-resolution structuralinformation available from mammography with the functional data acquired from MRI imaging. This paper presents a novel data fusiontechnique whereby medial–lateral oblique (MLO) and cranial–caudal (CC) mammograms (2D data) are registered to 3D contrast-enhanced MRI volumes. We utilise a combination of pharmacokinetic modelling, projection geometry, wavelet-based landmark detectionand thin-plate spline non-rigid ‘warping’ to transform the coordinates of regions of interest (ROIs) from the 2D mammograms to thespatial reference frame of the contrast-enhanced MRI volume. Of key importance is the use of a flexible wavelet-based feature extractiontechnique that enables feature correspondences to be robustly determined between the very different image characteristics of X-raymammography and MRI. An evaluation of the fusion framework is demonstrated with a series of clinical cases and a total of 14 patientexamples. 2003 Elsevier B.V. All rights reserved.

1 . Introduction tional) information provided by contrast-enhanced MRI.This is particularly useful in situations where small

This paper introduces a system that has been developed opacities, microcalcifications and fine spiculations presentto perform data fusion between 2D X-ray mammography in the X-ray image, cannot be correlated with CE-MRIand 3D contrast-enhanced MRI of the breast. The objective voxel enhancement. Although the 2D–3D fusion processhas been to develop a non-rigid registration and visualisa- uses novel techniques, including a robust wavelet-basedtion framework that makes use of both the high spatial landmark detection method and ‘pseudo X-ray’ imagesresolution of mammography and the physiological (func- generated from contrast-enhanced MRI, the actual non-

rigid technique can be considered standard in the literature.In many respects, this article can be considered a ‘clinical*Corresponding author.

E-mail address: [email protected](C.P. Behrenbruch). application’ paper.

1361-8415/03/$ – see front matter 2003 Elsevier B.V. All rights reserved.doi:10.1016/S1361-8415(03)00015-X

mailto:[email protected]

312 C.P. Behrenbruch et al. / Medical Image Analysis 7 (2003) 311–340

In general, there are several reasons for attempting to normally remains upright with the breasts compressed (onerelate mammograms to ‘functional’ images, typified by at a time), between two plates. All of the patients in ourcontrast-enhanced breast MRI. These include: study had both a medial–lateral oblique (MLO) and a(i) Complementary pathological indicators. For example, cranial–caudal (CC) mammogram, as is required for breast

microcalcifications present in a mammogram may cancer screening in the UK. It is worth noting that themeaningfully correlate with an enhancing region in angular separation between the mammogram view direc-MRI, despite the fact that calcifications arenot radi- tions depends on the woman’s size and shape, but typicallyologically visible in MRI. varies between 45 and 608. More importantly, as regards

(ii) Patients having a pre-operative MRI scan often already matching, the compression between the two mammogramshave a mammogram and therefore there is the oppor- differs significantly, often by as much as 1 cm.tunity not only to provide the clinician with more In order to register the projective X-ray image with thespecific information to support diagnosis but also to MRI volume, we project the MRI volume in both the CCmaintain diagnostic consistency across modalities. and the MLO direction. However, rather than simply use

(iii)Younger patients often have both an MRI scan and a voxel intensity or approximate T1-correlated X-ray at-mammogram due to the limited effectiveness of using tenuation characteristics(Gilhuijs, 1995), we use an en-mammography for assessing pre-menopausal women hancement measure based on a two-compartment phar-due to the X-ray attenuation characteristics of glandu- macokinetic model of contrast uptake(Tofts and Kermode,lar tissue. 1991; Hayton et al., 1997; Brix et al., 1991; Buckley et al.,

(iv) Post-operative tissue changes may be difficult to assess1994). This effectively produces a contrast-based projec-with mammography and are often better evaluated tion that has the visual and structural characteristics of anwith MRI. This is largely due to the high X-ray ‘enhanced’ X-ray mammogram.attenuation of scar tissue. The registration technique is outlined inFig. 1. It

The principal concept behind this work is the registra- actually consists of two separate registration stages. Thetion of a highly compressed 2D projective representation first ‘partial registration’ utilises a curvature measure toof the breast with an uncompressed volume-based acquisi- correlate boundary points along the film edge of thetion. In the case of the MRI acquisition, the patient lies mammograms with the edge of the volume projectionfacedown in the scanner with the breasts pendulous and(Marias et al., 1999).This curvature measure depends on auncompressed in the breast coils. This is entirely different good segmentation of the breast edge, which is achievedto the situation in mammography, where the patient by an intensity-based search, mathematical morphology for

Fig. 1. Overview of X-ray/MRI data fusion illustrating the two stages of the registration process.

C.P. Behrenbruch et al. / Medical Image Analysis 7 (2003) 311–340 313

Fig. 2. (A) The functional projection of the MRI volume in the cranial–caudal plane. (B) The deformation of the cranial–caudal X-ray of the breast to theMRI contrast volume projection (cropped). (C,D) The deformation grid of the partially and fully registered images. The increased level of complexity inthe deformation of (D) can be attributed to the internal landmarks used to complete the registration process.

smoothing and spline interpolation to produce a smoothed is appropriate in applications where we do not necessarilyedge profile. The effect of this partial registration is to have an intensity correlation between the MRI contrastdeform the extremities of the X-ray image to the boundary projection and the X-ray data and therefore intensity-basedshape of the MRI contrast projection. techniques(Horn and Schunck, 1981; Thirion, 1998;

A second registration is performed based on the selec-Christensen et al., 1994)do not generally work well. Thistion of internal landmarks using a wavelet-based feature approach also enables us to perform the partial registrationdetector. Internal landmarks in both the X-ray image and using only boundary points, which is particularly critical inthe MRI projection are matched using the following the case where the breast has undergone significant

1criteria: involution and there may be few landmarks or intensities• Scale localisation (i.e., landmarks of similar scale are suitable for controlling the internal deformation of the

matched). registration process. Typical registration (deformation)• Orientation. results of this two-stage registration process are shown in• Relative motion between the undeformed and the Fig. 2.

partially registered data from the first registration stage. In the following sections, the main components of the• Neighbourhood localisation in the partially registered registration process are discussed, namely the boundary

images. curvature measure, internal landmark detection and the useA wavelet-based technique has been used because of approximating thin-plate splines. Firstly, however, it is

features common to both the X-ray image and MRI beneficial to discuss the notion of compression compensa-projection may have a feature-space representation that has tion, which is critical to our choice of registration frame-dramatically different scale characteristics. For example, a work.lesion visible in an X-ray mammogram may have finespiculated details in addition to a main ‘blob’-like charac-teristic, which is more representative of the resolution of 2 . Compression compensationthe MR images. Additionally, there are many fine structur-al details in an X-ray image that do not have a feature The underlying challenge in the breast 2D–3D datacorrespondence with MRI and therefore a scale-sensitive fusion process is compensation for the compression of themethod that has some ‘adaptability’ is important. Wavelet mammographically imaged breast. This is necessary as the[packet] scale space with a best-basis function certainly MRI breast acquisition is not compressed (but is de-provides a convenient and relatively easy to implement formed/elongated by gravity) and therefore compensationsolution. The overall framework of the wavelet-based is necessary to enable features visible in the mammogramsmethod used in this paper is not new, although the idea of to be spatially correlated with the MRI volume.a feature-driven registration algorithm which has a notion Previous work related to modelling breast compressionof ‘feature scale’ rather than merely taking a multi-scale has used a cross-section approach to uncompress the breastimage-driven approach is novel. to enable point correspondence between mammogram

Registration completion involves using a combination of1The term ‘involution’ refers to the transformation of ductal ormatched internal landmarks and breast boundary points.

parenchymal breast tissue into fat. The incidence of involution increasesThe registration technique adopted to date is thin platesharply at the onset of the menopause. The transradiance of fat relative to

spline warping(Bookstein, 1989; Evans et al., 1991),a ductal /parenchymal tissue is the reason screening is typically restricted tovariation of radial basis function (RBF) elastic deformation women over the age of 50, a heuristic approximation to the age of onset(Arad and Reisfeld, 1995).This landmark-driven approach of the menopause.


Fig. 3. (A) A 3D shape reconstruction based on the two uncompressed mammogram views (elliptical volume interpolation between views). Due to the factthat both the CC and MLO planes are compressed, the shape appears somewhat cylindrical. (B) A shape reconstruction based on the technique described in(Kita et al., 1998).(C) The shape of the breast based on the two mammograms registered to the MRI volume projections.

MLO (medial–lateral oblique) and CC (cranial–caudal) technique of Kita et al. and a reconstruction based on theviews (Yam et al., 1999; Kita et al., 1998).However, in mammograms registered to the MRI functional projections.our approach, the objective is to uncompress the mammog- At first glance,Fig. 3(C), a two-view reconstructionrams into the reference frame of prone-acquired 3D MRI using the registered X-rays, appears somewhat unusual.volumes and therefore a different strategy is required. The For comparison,Figs. 4 and 5show the same registeredobjective is to use the ‘uncompression’ of the mammogram and reconstructed views together with the segmented MRIimages not only to facilitate a framework for point projections. Not only is there good shape correspondence,correspondence between mammograms, but also to enable but also it is possible to reconcile, albeit approximately,pathology that is visible in the mammogram to be com- the location of the pathology between the two differentpared with MRI. As a consequence, the uncompressed modalities.shape is very different to that proposed byKita et al. The registrations shown inFig. 4 can then be used to(1998) as the reconstructed shape resembles the prone- reconstruct the approximate shape of the breast andacquired MRI breast, rather than the idealised upright compared against the MRI volume.Fig. 5 shows such ashape of the breast. This is illustrated inFig. 3 where an comparison, where the location of the tumour in theuncompensated 3D shape reconstruction from an MLO and example used earlier is compared with the location of theCC mammogram is compared with the model-based tumour in the MRI volume segmented using a 3D im-

Fig. 4. (A,D) The original CC and MLO mammograms, respectively. (B,E) The segmented MRI functional projections. (C,F) The original mammogramsregistered to the MRI projections using the approach outlined inFig. 1.


Fig. 5. (A) The location of the tumour in the (re-sliced) MRI volume. (B) The shape reconstruction displayed inFig. 4(C). (C) The location of the mass inthe 3D reconstruction based on an average of the CC and MLO view tumour centroid.

plementation of(Hayton et al., 1997).This example is of compare the functional structure of tissues with the high-interest because it highlights a potential problem that must resolution structure of X-rays. The registration of X-rays tobe addressed in the compression compensation approach, the reference frame of MRI enables not only corre-namely involution. If the breast is highly involuted, then it spondence between mammogram views, but also approxi-may not be possible to include more than the breast mates correspondence with imaged tissue using MRI.boundary (edge) in the model, which will most likelyover-simplify the internal deformation of the breast. 3 .2. Projection formation

The main idea behind the projection step is to create an3 . Pharmacokinetic projection creation image from an MRI breast volume that is geometrically

similar to a mammographic projection, but which repre-3 .1. Overview sents the breast in an uncompressed shape. Rather than

simply using an intensity-based projection or some kind ofThe starting point for the fusion method introduced in transfer function that maps MRI T1 values to approximate

this paper involves the simulation of mammograms from tissue X-ray attenuation characteristics, we project thecontrast-enhanced MRI, to enable comparison of the voxel enhancement characteristics using a pharmacokineticsimulated results to actual mammograms. By geometrically model of gadolinium uptake. This concept is illustratedprojecting MRI voxel contrast-enhancement (phar- below, where low enhancement tissues are given a lowmacokinetic) information (Fig. 6) to a cranial–caudal and/ level of intensity in the ‘pseudo X-ray’ projection.or medial–lateral (oblique) view, and registering the The projected intensity from high-enhancement voxels ismammograms to the resulting projection, it is possible to given a correspondingly high value. To map the projection

Fig. 6. (A) The conceptual geometry behind the projection process (for a cranial–caudal view), where the MRI voxel intensities are mapped into atransformed intensity plane. (B) Some typical voxel enhancement graphs with the fitted pharmacokinetic model. The dashed line shows the effective resultof integrating the fitted model to produce a projection intensity that is biased to voxels with high enhancement.


Fig. 7. A two-compartment pharmacokinetic model with typical contrast curves for fat, parenchymal (glandular) tissue and enhancing regions of interest.M is the mass of contrast injected into the blood stream with respect to time.k andk are inter-compartment exchange rates andk is the contrastin 12 21 out

excretion rate.

intensities, we integrate the Hayton–Brady phar- • Fatty tissue receives a low integrated projection en-macokinetic model of contrast uptake(Hayton et al., 1997) hancement.for each voxel with respect to enhancement time. The • Parenchymal (glandular) tissue is given an enhancementvarious components of this pharmacokinetic model are level that spatially corresponds well with the intensitiesdepicted inFig. 7. that are observed in a mammogram image.

Eq. (1) shows the derived bolus model for Gd-DTPA • Rapidly enhancing regions of interest (ROIs) are givenuptake, which is fitted to the contrast data (via a Leven- extremely high relative contrast.berg–Marquardt algorithm) to produce a difference of These characteristics are demonstrated in the two cra-exponential parameterisation (seeFig. 7(b) for example nial–caudal examples shown inFig. 8. In the first pair, thecurve fittings). patient has no pathology (except for a few coarse calcifica-

tions); however, the glandular tissue is strongly pro-A 2bt 2at nounced in the mammogram due to ductal ectasia. These]]Conc(t)5 (exp 2 exp ). (1)a–b ‘exploded’ ducts are clearly visible in the contrast-en-

The parametersA, a andb characterise the exchange rates hanced MRI projection as a correspondingly high level ofk , k and k (Fig. 7(a)) although for the purposes of contrast with respect to the fatty background. In the second12 21 out

controlling the projection, we are primarily interested in example, the patient has a large, poorly differentiatedthe rate and amplitude of the voxel contrast enhancementtumour (outlined) that has a similar intensity to glandularprofile. tissue in the X-ray, but which clearly stands out as a

strongly enhancing region in the MRI projection. It isworth noting that the comparative resolution between the3 .3. Projection functional characteristicsprojections is quite different due to the spatial resolution ofthe MRI scan (particularly the number of slices). TheThe result of the pharmacokinetic projection is a func-projected intensity distribution also varies, as it is depen-tional projection of the MRI volume that has a number ofdent on the degree of involution in the breast.interesting characteristics. These are:

Fig. 8. Two examples comparing the CC mammogram to corresponding contrast-enhanced MRI projections. The first example (A,B) shows consistentshape characteristics of moderately enhancing parenchymal tissue, due to ductal ectasia. The second pair (C,D) compares the contrast of a large tumour inthe X-ray image with the corresponding functional projection.


3 .4. Alternative pharmacokinetic approaches around 10 min, it is unrealistic to expect that the patientwill remain completely motionless. This is a particular

Recently, we have considered alternative phar- problem as symptomatic women are often naturally tensemacokinetic models(Armitage et al., 2001).The most when imaging begins but relax several minutes into thesimplistic of these is a statistical comparison of the integral procedure, resulting not only in general body (and henceof voxel enhancement. Although this provides reasonable breast) movement but internal soft-tissue deformation duetissue differentiation, the lack of a physiologically mean- to pectoral muscle flexation. A number of research groupsingful model to constrain the analysis renders the model have developed approaches to this problem includinghighly susceptible to noise, particularly for low-enhancing techniques utilising (non-rigid) optical flow(Hayton et al.,fatty regions. 1997)and tensor splines (affine plus local deformation) as

The model that has, to date, provided satisfactory results a motion correction framework(Reuckert et al., 1999).Weis the Hayton–Brady model. As discussed previously, this currently utilise a 3D variant of the approach developed bymethod is based on a relatively conventional two-compart- Hayton et al., which uses mutual information as a voxelment exchange model. More complex models exist that similarity measure combined with a novel Bayesian regu-include additional exchange components, notably that larisation method to perform optical flow registration.proposed byTofts and Kermode (1991).It should benoted, however, that low temporal resolution of theenhancement acquisition (7–9 time points, both breasts4 . Breast boundary feature detectionimaged at approximately 1-min intervals) does not supportsystem identification with a more complex model repre- The detection of suitable landmarks for boundary de-sentation. Of course, improved temporal resolution of the formation between the X-ray mammogram and the MRIMRI acquisition sequence resulting from a small field of volume projection first involves segmenting the breastview (FOV) or a single breast acquisition would make edge. To achieve this, the X-ray images are binarycomplex models more applicable. In fact, such sequences thresholded at the first ‘valley’ of the intensity histogramare already widely in use. However, current clinical using a simple gradient /slope detector as shown inFig.judgement accords greater emphasis to spatial rather than9(a). This approach also works for the MRI projectiontemporal resolution and both breasts are always imaged in although it is essentially unnecessary as background (air)case the disease is multi-focal. voxels are easy to threshold prior to projection. Following

segmentation of the breast area we generate an eight-3 .5. Motion correction prior to projection connected component outline by using mathematical mor-

phology (Sallam and Bowyer, 1996)(closing followed byThere are two additional issues that require considera- dilation and subtraction from the original binary), shown in

tion in order to accurately compare the mammograms to Fig. 9(c).their corresponding MRI functional projections. The first is Although this method results in an approximation of themotion correction. In order to evaluate the voxel enhance- boundary, it has the virtue of being very fast and thement with respect to time, it is necessary to ensure that resulting outline is smooth (due to morphology) whiletissue motion is limited. Since the CE-MRI acquisition preserving the anatomical features of the original imageperiod (including pre-contrast agent acquisitions) takes (for example, the location of the nipple). Once the breast

Fig. 9. The steps to generate breast boundary segmentation involve (A) bimodal histogram thresholding, (B) binary image creation and (C) morphologytoproduce the breast outline. (D) The outline over the breast superimposed on the original image.


outline has been detected it is sampled and approximated ‘pseudo X-ray’ MRI projections. Our boundary detectioniwith a cubic splineg (s) that is parameterised along the technique has been reliably applied to many mammograms

edge (via a simple graph search) for every three points of (Marias et al., 2000)and is sensible to use in this fusionthe breast boundary. This parameterisation enables the approach due to the outline shape consistency between theboundary curvatureK(s) to be calculated for each point MRI projection and the true mammogram. However, wealong the spline: have found that this is not an adequate correction for breast

compression, so that internal landmarks are required tox ? y 2 x ? yss s s ss complete the compensation for complex internal soft tissue]]]]]K(s)5 . (2)2 2 deformation.x 1 ys s

Previous work with mammography registration has fo-cused predominantly on aligning the boundary of the 5 . Partial registration from the breast boundarybreast in each mammogram(Marias et al., 1999; Sallamand Bowyer, 1996; Wirth and Choi, 2000).In our applica- 5 .1. Overviewtion, we automatically calculate the location of the nippleand the points between the breast, the rib cage and the Radial basis functions (RBFs) are used for the elasticaxilla (in both the mammogram and the projection) on the image deformation in this registration scheme. RBFs arebasis of separating positive and negative curvature portions used in two contexts—firstly, in an interpolation frame-of the spline parameterisation and subsequently locate work for the boundary conditions where the regularisationcurvature peaks as shown inFig. 10. Separation of the requirements are simple. Later when internal landmarkspositive and negative curvature components from Eq. (2) are included, information about the spatial characteristicsis very important as it enables the detection of consistent of the deformation points is used to implement a morelandmarks near the rib and the axilla. sophisticated regularisation that is based on an approxi-

These curvature-selected points form the basis of the mation method. We start with the interpolation case as it isfirst step in the registration process, which effectively more general to defining a RBF.ensures that the boundary of the X-ray image is warpedonto the MRI volume projection. A non-rigid transforma- 5 .2. Interpolative radial basis functionstion is made possible by interval subsampling at regularintervals between the curvature-detected landmarks (5–10 In RBF interpolation, a set ofn landmarks (p ,q ) isi i

2 2points per segment, depending on the size of the breast), in used to define a transformation functionu:R → R whereorder to generate a large number of points for ‘warping’ p 5 (x , y ) are the landmarks in the first image andi i1 i1

the boundary between the X-ray mammograms and theq 5 (x , y ) are the landmarks in the second. An interpo-i i2 i2

Fig. 10. The curvature profile along the breast boundaryK(s), consisting of the two peaks near both ends of the boundary and the peak corresponding tothe nipple in the middle. The boundary is then sub-sampled at intervals between the rib-nipple and nipple-axilla segment to produce enough points to‘warp’ the boundaries.


¨lation transformation functionu(x) based on point land- 1997),inverse multi-quadrics (R ) (Ruprecht and Muller,IM

marks must fulfil the following constraint: 1993)and Gaussian RBFs (R ) (Arad and Reisfeld, 1995):G

42d 2u( p )5 q , i 5 1, . . . ,n, (3)i i r ln r , 42 d [2N,R (r)5 (10)HTPS 42d2 r , otherwise,and minimise a functionalJ( f ) for R .

2 2 m2 2 2 2 2 2 R (r)5 (r 1 c ) , m [R , (11)≠ f ≠ f ≠ f M 12 ] ] ]]J ( f )5EE 1 1 2 dx dy,S D S DH S D J2 2 2 ≠x ? ≠y≠x ≠y2 2 2 2mR R (r)5 (r 1 c ) , m [R , (12)IM 1

(4)2 22r / 2sR (r)5e . (13)Gwhere this functional is also known as the bending energy

of the deformation. In this case we are solving the In our work to date, we have used the thin-plate spline2biharmonic equationD f 5 0 whereby local deformation radial basis functionR (10), since the effect of de-TPS

conforms to a functionf(x,y) for minimisation of the formation is both intuitively reasonable and (qualitatively)bending energy. For the interpolation case, each coordinateproduces results that appear clinically consistent. Controlis calculated independently (i.e., the interpolation of regularisation is extended in the second stage of the

du :R → R is solved for each coordinatek 5 1 . . .d, in thisk registration process, but it is conceivable that with propercase d 52, with the corresponding constraintsu ( p )5k i parameterisation of the distance functions, it would beq ,k). In the general RBF approach, the interpolationi possible to use other RBFs (for example, a Gaussian) infunction u(x) consists of two parts: this application.

u(x)5f (x)1R (x), (5)s s

wheref (x) is a sum of polynomials up to degreep and 6 . Wavelet-based internal landmark detections

R (x) consists of a sum of RBFs:s

6 .1. OverviewM

f (x)5O b f (x), (6)s j jj51 To complete the registration process, corresponding

internal landmarks are detected in the registration pair and,N

along with various matching conditions (discussed in theR (x)5O a R ix 2 p i . (7)s ds i ii51 next section), are used to compensate for the complex

internal deformation of the breast. We cannot rely on aThe f (x) are a separable basis ofM functions for allj field-based (intensity-based) registration technique as thepolynomials up to degreep. R(r)5R(iri) is a distancedeformation may be quite large and the local textural detailfunction (depending only onr > 0 from the origin) thatof features is different in the X-ray and correspondingdescribes the local deformation (interpolation) characteris-CE-MRI projection. Therefore it is necessary to implementtics of points around the RBF landmarks. In Eq. (7),a robust feature detection algorithm, which in turnix 2 p i5 iri is the Euclidian distance fromx to p anda ,i i I produces landmarks that can be matched and used as the

b are coefficients, with the RBFsR(ix 2 p i) centred onj i basis for non-rigid registration refinement.the N landmarksp . Combining (6) and (7) and using thei Various spatial filtering techniques based on localiseddeformation constraint:Fourier operators and trigonometric transforms have been

nused for image feature detection(D’Astous and Jernigan,O a f p 5 0, j 51, . . . ,M, (8)s di i i 1984; Haralick et al., 1973; Turner, 1986; Bovik et al.,i51

1990).The inherent limitation of these approaches lies inyields a system of linear equations for the coefficientsa, the need to define a large number of Fourier windows (in ab : multi-resolution sense) in order to produce a maximally

sensitive feature detector. In order for these techniques toK P a qk5 , (9)T work robustly, there is a need to be able to adapt theS DS D S DP 0 b 0

spatial localisation properties of filters as meaningfulwhereK is then 3 n matrix K 5R(ip 2 p i) andP is the descriptors with respect to image content. Most windowedij i j

TN 3M matrix P 5f ( p ). The q 5 (q , . . . ,q ) are Fourier /Gabor techniques can do this only to a limitedij j i k k,1 k,n

the set of target landmarks. extent.There are a wide variety of radial basis functions used in An alternative to windowed Fourier techniques is the

the literature for elastic deformation of images. These discrete wavelet transform (DWT), a dyadic filteringinclude thin-plate splines (R ) (Bookstein, 1989; Rohr et implementation that provides a true multiscale foundationTPS

al., 1996),multiquadrics (R ) (Hardy, 1981; Little et al., for image analysis(Mallat, 1998).A useful variant of theM


DWT is ‘discrete wavelet packet analysis’ (DWPA)(Coif- identified with wavelet generation functionf in (Mallat,man et al., 1991, 1992)which is essentially a reconfigura- 1989).The functionc is the ‘mother wavelet’ associated1

tion of the basis subspaces. This results in particular linear by convolution withH and G. The descendent functionscombinations (superposition) of wavelets, producing a (tree ‘leaves’)c are analogous to iterated forms of then

library of wavelet projections (called ‘packets’) with which mother wavelet as in the DWT; however, in this case, theto analyse a function. Additionally, DWPA forms sub- evolution ofc is over the complete tree. The collection ofn

spaces that retain the orthogonality, smoothness and locali- these functions forn 5 0,1,. . . ,m makes up the waveletsation properties of the dyadic DWT, but with better packets associated with a particular choice of quadraturespatial localisation characteristics. filters. It is also apparent that in this way, wavelet packet

The improved spatial characteristics are demonstrated in analysis is a generalisation that includes the DWT de-an example inFig. 11, where an (approximate) impulse compositionfunction is decomposed using both wavelets and wavelet The most straightforward way to use wavelet packets ispackets. When viewed as a tiling of information cells in a fixed scale context. In this way, they are somewhatacross the space-frequency plane, the DWT analysis analogous to analysing a function or sequence with a DWTcorrectly localises the peak of the function in the high at a fixed scale or level, albeit with a different set of filterfrequency elements, but is forced to include poorly local- projections. In our case, we use wavelet packets in aised, low frequency components as well. Although the multiscale context, which effectively means that the ap-analysis equivalent for the wavelet packet case does not proximation can be performed using projections anywherecontain all spatial-frequency tilings, it demonstrates that it in the decomposed basis. For further details of the conceptsis possible to reconstruct the signal using almost a single of wavelet analysis and scale-space techniques, we refer towavelet packet. For this reason, we have used DWPA as(Witkin, 1983; Burt and Adelson, 1983; Daubechies et al.,the foundation of our feature detection, particularly as we 1986; Stein, 1979; Marr, 1982; Grossman and Morlet,are interested in highly localised features such as calcifica-1985; Mallat, 1998)and for a very useful overview oftions and spicules are good quality, consistent landmarks signal processing with wavelet packets (as well as a goodfor registration. treatment of the software implementation issues)(Wicker-

Using H andG to represent the equivalent high and low hauser, 1994).pass filter components of a set of conjugate quadraturefilters, the following recursive set of functions can be 6 .2. Best basis selectiondefined:

The multiscale wavelet packet representation provides adef

rich set of filtered projections within a wavelet basis atc 5Hc ; E c (t) dt 5 1, (14)0 0 0

different scales. Using wavelet packets for feature de-R tection involves ordering these subspaces by information

def ] content to produce a scale-space ‘stack’ for analysis. ThisŒc 5Hc ; c (t)5 2O h( j)c (2t 2 j), (15)2n n 2n n approach is called ‘best basis analysis’(Coifman andj[Z

Wickerhauser, 1992).Before constructing such a scaledef ]Œc 5Gc ; c (t)5 2O g( j)c (2t 2 j). (16) space, it is necessary to define the notion of an information2n11 n 2n11 nj[Z cost function that is used to rank each representation in

terms of the contribution to approximating a sequence. ToThe functionc (the ‘root’) is uniquely defined and is0

Fig. 11. A comparison between DWT and DWPA representation of an (approximate) impulse function. The DWT case (left) correctly determines thelocation of the peak for the high-frequency components but also includes low frequency components with poor spatial localisation. The DWPA case (right)demonstrates much better spatial localisation and produces a more optimal representation of the impulse.


do this, we use a real-valued functionalM, satisfying the (Pincus, 1992, 1997).The idea is that wavelet packetcondition decompositions that smoothly approximate the original

sequence will have a higher degree of regularity, whilstM(u)5O m uu(k)u ; m(0)5 0. (17)s d equivalent high-pass filtered components that reflect the

k[Zsignal noise and will be more fluctuating in nature,

Here m is a real-valued function defined on [0,~). If we exhibiting greater irregularity. ApEn is defined as follows:assume thato m(uu(k)u) converges absolutely, thenM willk

N2m11be invariant to rearrangement of the representation of the 1m m]]]F (r)5 O log C (r), (23)isequenceu. We can then define a functionalM on the setx N 2m 1 1 i51of basesB for eachx [X:

where given a positive integerN and nonnegative integerM : B → R; B → M(B*x), (18)x m, with m <N, a positive real numberr, and a sequence of

real numbersu:5 (u(1),u(2), . . . ,u(N)). The distance be-which is referred to as theM-information cost of atween two sequencesx(i) andx( j), wherex(i)5 (u(i),u(i 1sequencex in the basisB. Another way of looking at this,1), . . . ,u(i 1m 21)) is defined by d(x(i),x( j))5is to evaluate a transformation off(x) to g(x) via an

mmax (uu(i 1 p 2 1)2 u( j 1 p 21)u). C (r) is theoperator (in this case convolution–decimation) as a dis- p51,2, . . . ,m i

number of j <N 2m 11 such thatd(x(i),x( j))< r, nor-tance measure with respect to a transformation parametermalised with respect toN 2m 1 1. Thus, ApEn(m,r,N)(u)l.may be interpreted as a measure of the frequency at which2i f 2 gi 1ligi, (19)L (I )2 number sequences of lengthm occur compared withsequences of lengthm 1 1. High values of the ApEnevaluated asmeasure imply a highly random nature, whereas low

1 / 2D2 2 values suggest a relatively smooth sequence.i f 2 gi 5 E u f(x)2 g(x)u dx . (20)L (I )2 S D Therefore, if we apply the ApEn measure to each

Iwavelet packet decomposition of a sequence, we can order

This enables each approximated component of the waveleteach projection by the contribution it makes to completelypacket decomposition to be compared and subsequentlyreconstructing the sequence. Starting from the lowestranked by information content. There are many different approximate entropy packet decompositions and pro-kinds of information cost functionals that can be used. For gressively adding detail (higher ApEn), one can construct aexample, a simple threshold́ based on coefficients in a scale-space using multi-scale wavelet packets, despite thewavelet subspace could be used to order the contributionfact that different scales (depending on whether they areof each packet to the complete reconstruction of a se- equivalent low-pass or high-pass filtering combinations)quence, as shown in Eq. (21). may be ranked counter-intuitively by virtue of the level of

decomposition. An example of such a scale-space is shownuv u, if uv u>´,in Fig. 12, where a small portion of an X-ray image ism(v)5 (21)H0, if uv u,´. decomposed into a wavelet packet scale-space stack (2Dsequences are discussed more thoroughly in the nextAnother useful approach is the entropy of a representation,section). It is worth noting that the resulting scale spacesuch that a cost functionu 5 hu(k)j is evaluated byappears similar to many other ‘scaleogram’ techniques,

1 including phase-based methods such as(Kovesi, 1996).]]H(u)5O p(k) log , (22)p(k)k The difference with our method lies in the fact that2 2 best-basis selection pre-orders the scale decomposition in awhere p(k)5 uu(k)u / iui is the normalised energy of the

fashion that is more oriented to the feature content of thekth element of the sequence. This functional can beimage (by virtue of using morphologically relevantevaluated in a minimisation context and used to rank a setwavelets). The result is a feature-driven scale space thatof decompositions.dramatically improves the ease of defining salient regions.One of the difficulties with using conventional entropy

measurements is defining a probability density functionpin a way that is appropriate to each representation. 6 .3. Choice of wavelet packetPractically, this means selecting a histogram samplingdensity that meaningfully captures the statistical charac- Because of the superposition of wavelets, waveletteristics of the sequence. Clearly, in a multi-scale context packets provide a large range of multi-scale representationwhere this distribution may vary greatly (and with different with which to create a scale space. For this work, we havesampling resolutions by virtue of decimation), this is not a chosen a well-known class of existing wavelet bases,trivial task. To circumvent this problem, we have chosen to called ‘Coiflets’, which have a number of useful propertiesuse an approximation to entropy—Approximate Entropy that make them ideal for detecting regions of interest in(ApEn), which is a useful measure of signal regularity medical images. In particular, we wanted a wavelet basis


Fig. 12. An example scale-space illustrating cumulative addition of image detail (via packet reconstructions) using an information cost function.Thebottom of the scale-space contains a quite smoothed approximation to the image, whilst the top of the stack is a perfectly reconstructed image (i.e., allpackets have been used).

that would be sensitive to sharp, spatially localised features kets have broad frequency characteristics making themsuch as tumour spiculations, microcalcifications (in X-ray useful for noise approximation. These properties makeimages) and small regions of focal enhancement in MRI Coiflets very good for generating the kind of scale-space(such as for ductal carcinoma). Coiflets have the following shown inFig. 12,as a good spread of detail is ascribed touseful properties: both the smoothly approximated region of the scale-space• They have a high number of vanishing moments for a and the high-frequency (noise) components. In the case of

given support length (i.e., good frequency localisation features that have a broad Fourier localisation (for exam-as well as spatial localisation). ple, sharp changes in intensity due to edges or highly

• They are almost symmetrical. localised features), the Coiflet scale-space provides good• They have compact support. discrimination from smoother components. Some example• They are orthogonal (indeed a bi-orthogonal form is Coiflet wavelet packets are shown inFig. 13 (generation

also available). and scaling functions shown, as well as example higher-Low-order Coiflet wavelet packets are useful for approx- order packets). For more information on the generation and

imating ‘bumps’ without over-smoothing, hence they have properties of Coiflets we refer to (Daubechies, 1992).good edge-preservation characteristics. Higher-order pac- Additionally we are considering the development of cus-

Fig. 13. Example Coiflet wavelet packets (C2) including the generator /scaling functions (W0/1) and higher order members.


1 dtomised biorthogonal wavelets based on the ‘lifting (F ^ ? ? ? ^F )u(x)scheme’(Sweldens, 1996),which could be tuned to the def

5O ? ? ?O f (k ) . . . f (k )u(2x 2 k , . . . ,2x 2 k1 1 d d 1 1 d dmorphology of specific features or feature characteristics.k k1 dSuch methods could be used go produce customised

1 2 dwavelets for detecting specific features such as calcifica- 5F F . . . F u(x). (27)x x x1 2 d

tions, which have quite well defined image statisticalcharacteristics.

A visualisation of Eq. (27) ford 52 is shown inFig. 14.This approach is analogous to subband coding, a rela-

6 .4. Extension of wavelet packets to images (2D)tively conventional approach to filter decomposition. Interms of implementation, this means that the image is

Our treatment of wavelets, wavelet packets and scale-convolved (and decimated) with the quadrature filters in

space construction via information cost functions (besteach dimension in a ‘quadtree’ fashion. As an aside note,

basis analysis) has so far been limited to the basic 1D case.the actual implementation of tensor wavelets requires some

However, for analysing images, these concepts need to becare with respect to the issue of mixing scales via tensor

extended to higher dimensions. Fortunately, most conju-products(Mallat, 1998).

gate quadrature filtering algorithms extend quite neatly tohigher dimensions via the tensor product of 1D basiselements:

6 .5. Feature extractionb(x)5 b(x , . . . ,x )5 b (x ) ? ? ? b (x ). (24)1 d 1 1 d d

Now that we have defined our notion of a scale-space inThis tensor product basis approach is termed ‘separable’terms of ‘best-basis’ ordered reconstructions from waveletbecause they can be factored across integrals to obtain apackets, it is possible to perform feature extraction. Thesequence ofd one-dimensional problems by treating eachextraction process involves region growing the scale-spacedimension individually. The equivalent low- and high-passand identifying regions that persist through many levels offilters H and G in the definition of the wavelet packetscale.decomposition are quadrature filters with corresponding

Unlike the ‘extremum’ stack concept ofKoenderinkdual (adjoint) filters that can be used for reconstruction in(1984) or the conventional CWT scaleogramthe same as the 1D case. Therefore the tensor product(Holschneider, 1995),our scale-space construction doesapproach to extend the wavelet packet analysis to highernot guarantee continuity through the entire stack by virtuedimensions naturally involves the tensor product of 1Dof our use of an information cost function to order thesequences (filter coefficients),h andg. The tensor productwavelet packet construction. Therefore we are interested inof quadrature filters on the bivariate sequenceu 5 u(x,y) isregions that may be well localised across several scales,defined asbut not necessarily in a congruent fashion acrossall scales.

H ^G u(x,y)5O h(i)g( j)u(2x 2 i,2y 2 j)s d The most significantn regions are treated as ‘features’ andi, j are labelled by the highest level of scale reached by region

growing/merging. A visualisation of this effect is illus-5O h(2x 2 i)g(2y 2 j)u(i, j). (25)i, j trated inFig. 15 where a small portion of an X-ray image

containing microcalcifications (sharp, high-intensity fea-Similarly, for the adjoint (reconstruction) case, the tensortures) is visible as distinct regions propagating through theproduct (H ^G)* ;H* ^G* is defined bystack.] ](H* ^G*)u(x,y)5OO h(2x 2 i)g(2y 2 j)u(i, j). (26) The region growing starts with a collection of seed

i jpixels at the low-scale part of the stack. A merging

These definitions generalise to an arbitrary dimensiond for operator is applied to adjacent pixelsl , l , l , li11 i21 j11 j21ia filter operatorF [ hH,Gj for i 5 1, . . .d: and l , l in the scale-stack volumeV(x,y,l) usingk21 k11

Fig. 14. The implementation of wavelet packet analysis ford 5 2 is performed by directional convolution and decimation for each dimension of the image.


Fig. 15. A small section (1403140 pixels) of a 30-mm digitised mammogram containing two calcifications is decomposed into the wavelet scale-space.The calcifications are detectable as a well-localised propagation through the sliced scale-space (C and C ). Note: the scale-space has been inverted to1 2

better show the shape preservation from coarse to fine scales.

significant regions) or the number of regions converges. InO V(n) O V(n) our experimentation, we have found that it is better to useL L Li j k n[X n[Y]]]] ]]] ]]]DE 5 2 fewer good quality landmarks, therefore this numberm can**L 1 L 1L L Li j k i j either be controlled interactively by the user (typically

2 m , 10) or the landmark selection criteria (described in theO V(n)n[L next section) used to reject features that have less signifi-]]]2 2G (28)**Lk cant scale localisation. This results in a segmentation of the

scale-space that can be visualised as an overlay on theand

image under analysis (Fig. 16).,0,l ,l ,l merge, Once the regions have been segmented they are fittedi j k

DE (29)H with a cubic spline contour and the image labelled into>0,l ,l ,l unchanged,i j kgeneralised areas of ‘region of interest’ and ‘noise’. The

where enables a histogram of pixel values to be assigned to‘features’ or regions of interest (ROIs; up to a certain

L 5O L(n), L 5O L(n), L 5O L(n). (30)i j k scale) and ‘noise’, which has a different statistical dis-n[l n[l n[li j k

tribution of intensities. With these estimations for featureand noise, we utilise the region competition algorithm ofL(n)5 1 and L gives the vector of intensities ofl . Thisi i

Zhu and Yuille (1996) to slightly refine the contourprocess is iteratively repeated for a fixed threshold valueG

boundaries of the detected features, as a small amount ofuntil no more merges are possible (i.e., the region issmoothing has transpired during the approximation pro-completely grown).G is then increased until either thecess. This refinement step is not strictly necessary, al-desired number of regions is produced (them most

Fig. 16. Segmentation results for the small region of a mammogram illustrated in the previous figure. There are several features in this image (left)including a small curvilinear structure and two clusters of calcifications (circled). The segmentation (right) has produced several regions of interest whilstbeing relatively impervious to the noisy characteristics of the image.


Fig. 17. An example of wavelet-based feature detection and segmentation of a mammogram of a patient with very dense breasts. This is a good example asit shows image scale-space can be used to produce a limited number of high quality landmarks representing visually salient features (breast edge segmentedearlier in the process).

though it produces feature contours that have a higher on the feature contour to adjust the sensitivity of thedegree of complexity and better reflect the actual morphol- match to ‘blobs’ and curvilinear structures.ogy (for example the fine spiculations of a tumour). An • Feature area. This does not have the same meaning asexample of a completely segmented mammogram image is the scale localisation parameter, although there may beshown in Fig. 17 (minus the breast edge which is seg- some overlap in the case of large features. The objec-mented earlier). It is this collection of features that is used tive is to ensure that spatially localised features are notas the basis of the internal landmark matching. matched to larger features. This is parameterised by a

In terms of actually defining landmarks for the RBF fixed threshold (we have found that allowing up to 30%deformation, we ignore the pectoral muscle region and the variability provides a reasonable result).radiological tag and since our features are based on closed• Relative motion. Motion between the landmarks in thecontours grown from the scale-space stack, RBF land- undeformed and partially registered (via boundarymarks are based on the centroid of each feature. Clearly curvature) data. This is illustrated inFig. 18,where thethis introduces a degree of error into the registration; relative motion of landmarks is included as a weightinghowever, the approximation framework largely compen- factor in the match.sates for this in the completion registration step and will be • Neighbourhood localisation. A window proportional toalso influenced by the number of landmarks used. the amount of displacement of the transformed internal

landmarks from the boundary registration is used tolimit matches to a ‘window’ or neighbourhood.

7 . Landmark matching All of these parameters are used to drive a simple spatialsearching algorithm that is conceptually similar to the

Given a set of segmented landmarks (represented in the iterated closest point method (ICP)(Besl and McKay,deformation framework by a feature centroid), we compute 1992; Declerck et al., 1996).Essentially these criteria area correspondence measure between features in the registra- used as the basis of a match rejection filter to ensure thattion pair on the basis of the following criteria: landmarks classified as a ‘match’ have similar spatial• Scale localisation (also used to parameterise Eq. (31)). properties. The search characteristics are then also adjusted

This reflects the comparative morphology of landmarks on the basis of how landmarks have demonstrated a changeas features with similar textural characteristics will be in geometrical correspondence as a result of the initialclosely related in wavelet scale-space. boundary deformation—in other words, how the initial

• Orientation. We compute a simple orientation measure registration process improves alignment between internal


Fig. 18. This figure illustrates how the boundary registration is included as rejection criteria for the matching algorithm. (A) The pre-registrationrelationship between three points in pair of images to be registered (transformation5identity) with three matches classified on the basis of a closest-pointsearch. D and D represent distance measures between candidate pairs of points. After the boundary registration (B), the candidate pair corresponding to1 2

9D is maintained, despite a small change in the distance measure (thresholded). However the remaining landmarks are re-matched on the basis of the1

transformationT .Boundary

features. This partial registration search refinement algo- we restrict our matching on the basis of features withrithm is outlined inFig. 18. similar scale localisation in the wavelet ‘feature stack’.

This effectively means that we are not interested inmatching features that change size (due to differences in

8 . Completion of registration using internal breast compression, for example) or in the case of multi-landmarks modal fusion, features that may have a different spatial

representation between modalities.8 .1. Overview To implement this in a registration algorithm in such a

way that landmark localisation errors can be accommo-The last step in the registration process is to include dated, it is necessary to weaken the interpolation condition

both the boundary curvature-based landmarks and detected in such a way that the resulting transformation is ainternal landmarks in the registration process (as outlined compromise between smoothness and data adaptation. Toin Fig. 1). However, in this stage of the registration achieve this, we define a new functional (refer to Eq. (4))process, we use an approximation rather than an interpola-J (t) to be minimised as a function ofl. This l is al

tion approach to the elastic deformation. This is to account regularisation parameter that controls the balance betweenfor some of the inaccuracy of centroid-based landmark smoothness and approximation(Rohr et al., 1996):delineation as well as to produce a smooth deformation

n 2q 2 f( p )i i 2that honours landmarks to which we ascribe different ]]]J ( f )5O 1l ? J ( f ). (31)U Ul 2 2sdegrees of confidence by virtue of wavelet scale localisa- i51 i

tion properties. 2The scale localisation information is used to control thes i

terms that have the effect of weighting the smoothness in8 .2. Approximating radial basis functionsthe deformation by the saliency of the landmark. In thisway, greater deformation influence can be attributed toFor the boundary registration case, we have relativelyfeatures that have stronger spatial localisation. The solu-high confidence in our landmark selection since it is basedtion process is very similar to the interpolation problemon a robust geometrical measure (i.e., curvature) that hassince only theK submatrix in Eq. (9) needs to be changed:relatively little ambiguity. However, in the case of addi-

21tional internal landmarks, it is important to consider the K95K 1l ?V , (32)level of confidence we can place on our selection and

2 2evaluation of corresponding features. whereV5diagh1/s , . . . ,1 /s j. A simple example of the1 n

In the wavelet feature detection and subsequent feature effect of varyings is shown in Fig. 19(C), where thematching, we have chosen a set of criteria that limits the relative weighting of landmarks produces a smooth ap-error in search for feature correspondence. In particular, proximation.


Fig. 19. The regularisation approach of the second step of the registration process. (A) Contains five landmarks (1–5) in the undeformed image. In (B),landmark[3 is moved and the resulting deformation approximation calculated assuming that thes weighting for the remaining landmarks are five timesgreater in the transformation calculation. (C) The case where landmarks[1,2,4,5 are fixed and landmark[3 moves. The transformation interpolates alllandmarks equally, with a highly deformed result (i.e., the elastic deformation is too global despite the fact that only one landmark moved). (D) The casewhere all landmarks are used in the approximation process, but thel set (high) so that the transformation is smooth.

9 . Clinical applications potential to further improve the effectiveness of screening(Karssemeijer et al., 1998; Highnam and Brady, 1999).

9 .1. Overview However, for younger women (typically equated withpre-menopausal), women with very dense breasts and

Before presenting the results of data fusion for a number patients who have radiologically exhibited tissue regenera-of patient studies, we present three particular cases that tion as a result of hormone replacement therapy, mammo-motivated this work. These case studies are quite specific graphy has been proven to be less reliable(Kavanagh etin the sense that they relate to particular pathological al., 2000; Leung et al., 1997; Litherland et al., 1997).conditions; however, the utilization of our data fusion There is even some evidence to suggest that MRI-basedframework can be generalised to difficulties with imaging screening may be a more appropriate first-line approach tothe breast in certain circumstances. These studies are also accurate detection and diagnosis for these cases (par-of interest from a validation perspective, because in each ticularly if a patient falls into a high-risk category)case, a wire localisation or needle biopsy was used to (Warren and Hayes, 2000).confirm the location of the pathology. In the first example, a 47-year-old woman presented

with a lump that was locally painful and marginally9 .2. Case [1: mammographic imaging of women with palpable under a clinical examination, but with diagnosti-dense breasts cally ambiguous mammograms due to the density of the

breast tissue. Two attempts at core biopsy, both withThere is little contention about the screening value of mammography and with ultrasound guidance, were unsuc-

mammography in the early detection and diagnosis of cessful in localising the lesion. As a frustrated attempt tobreast cancer. Mammographic screening has greatly im- obtain more information, the patient was referred for aproved the mortality statistics(Fletcher et al., 1993; CE-MRI scan. The results of the mammographic imagingNystrom et al., 1993; Verbeek et al., 1984),while emerging and a relevant slice of the corresponding MRI acquisitiontechnologies such as computer-aided detection (CAD) are shown inFig. 20. In this case an extended CCcombined with full-field digital mammography have the mammogram was taken in the hope of spotting a lesion


Fig. 20. Initial assessment of the mammograms with the suspected ROI highlighted in the lateral mammogram (centre) along with a corresponding regionin a slice of the CE-MRI volume (right). No lesion was obviously apparent in the CC mammogram (left).

closer to the axilla as the conventional CC did not reveal location of the pathology when compared with the clinicalany suspicious regions. The clinician’s estimation of the assessment of the original mammograms. In this case, theROI in the lateral view (used for biopsy) and the corre- patient was imaged retrospectively using MRI and asponding assessment of the MRI are highlighted. localiser inserted. The patient mammograms were then

The lesion is clearly visible in the MRI acquisition as a retaken with the localisation wire to confirm the registeredmoderate-enhancement circular morphology. After regis- estimation of the ROI. A subsequent biopsy tested positivetration with the corresponding CC (extended) and MLO for malignancy. The actual location of the pathology in themammograms (Fig. 21), the location (centroid) of the ROI ex-CC and MLO view is shown below.in the MRI volume was transformed back to the two From a clinical point of view, the patient managementmammogram views and presented to the clinician as a was difficult in this case. A long period elapsed betweenmarker (Fig. 22). the initial examination and the final positive needle biopsy.

The projection of the pathology is quite successful in the Though this was partially due to the number of imaginglateral view, although the CC view (minus the bright stages required to confirm the diagnosis, the ambiguity ofregion corresponding to the pectoral muscle in the MRI indicators present in the radiological assessment was awhich is not present in a CC mammogram) is obscured by major contributing factor. Naturally, this created bothan accumulation of glandular enhancement (a small clinical pressure and patient stress that might have beenamount of grey-level rescaling helps somewhat). The alleviated had the diagnostic indicators been less complex.inverse transformation of a clinician-selected point in the Although the additional MRI and post-localisation mam-MRI volume produced a slightly different suggested mograms were possibly an unnecessary process, a robust

Fig. 21. Registration results between both mammogram views and the MRI projection with corresponding ROI location highlighted.


Fig. 22. The initial clinical assessment of the extended CC mammogram (left) did not reveal any suspicious regions of interest. The MLO mammogram(right) was determined to have a slightly denser region (circled) but did not reveal any pathology with needle biopsy. After registration with the CE-MRIprojections, the centroid of the region visible in the MRI was projected back onto the mammograms (indicated by1). Biopsy of this region tested positivefor malignancy.

implementation of our fusion approach could eventually cancer and hence the limited resolution of MRI (and partialfacilitate this kind of comparison at the time of the MRI volume effects), which further compounds the issue.acquisition. In this example, a young (26 years of age) woman

presented for a routine mammogram screening which9 .3. Case 2: ductal carcinoma in situ (DCIS) revealed three distinct clusters of calcifications largelycharacterised by microcalcifications visible in both views. Normally, the age of the patient

would preclude a mammogram, however she had alreadyDuctal carcinoma in situ (DCIS) is a commonly occur- had an occurrence of ductal carcinoma 4 years previously,

ring form of breast cancer that can be difficult to assess which was treated by a wide local excision in the uppermammographically. A clear indicator of DCIS is often the (outer) quadrant of the breast followed by radiotherapy.presence of calcifications with a ‘jagged’ morphology Previous surgery also adds complication to mammography(Lanyi, 1996; Wolfe, 1982),which are usually quite clearly imaging as surgical scarring effectively masks pathologyvisible in at least one of the mammogram views. A (scar tissue has unfavourable X-ray projection characteris-difficulty with DCIS is the often impalpable nature of the tics) and therefore the patient was sent for a follow-uptumour during clinical examination. From a surgical MRI. However, the clinicians were interested to know theperspective, therefore, 3D localisation and visualisation is correspondence between the calcification clusters visible invery useful. This is because of the high dependency on the the mammogram (despite the dense tissue) and the CE-palpability of the tumour to guide the resection process. An MRI acquisition.Fig. 23 depicts 4 years of mammogramsobvious way of localising the tumour in 3D would be a (both views) that have been registered using the techniqueCE-MRI acquisition. described in(Marias et al., 2000).In this way, it is

Unfortunately, the sensitivity of CE-MRI to DCIS is possible to compare the changes in tissue density due tovery low due to the relatively small amount of tumour both the surgical treatment (the excision was followed by avascularity and hence the contrast-enhancement profile of breast reconstruction using muscular tissue which is clearlythe pathology is usually too subtle for reliable detection evident in the sequence) and possible evolution of pathol-(Rankin, 2000).However, if the locations of the calcifica- ogy.tions in the mammogram views could be transformed to This first stage analysis of the mammographic history ofthe spatial coordinates of the MRI volume, it might be the patient was a useful exercise. Firstly, the surgeon feltpossible to prompt the radiologist with regions of interest that the calcification cluster located near the previousthat should be more clearly scrutinised for suspicious excision site (cluster labelled ‘1’) was a relatively low-risk(although subtle) enhancement and morphology. Of course, of cancer since surgical scarring may produce a slightan additional complication with imaging this kind of dusting of calcifications. Certainly the shape and dis-pathology using MRI is the relatively small size of the tribution characteristics of the microcalcifications were


Fig. 23. Four years of patient mammograms registered using the techniques described in(Marias et al., 2000).At year 4, three clusters of calcifications arevisible in both the CC (top row) and MLO (bottom row) mammograms. Comparing the location of cluster ‘1’ to the surgical excision area (circled regionsin the mammograms of year 2) suggests calcification due to surgical scarring rather than malignancy.

consistent with this expectation. The second cluster of previous figure are subtle, there was enough quantitativecalcifications (labelled as ‘2’) was somewhat contentious evidence (from the contrast profile) to indicate that thedue to the location, particularly with respect to the nipple suggested location of the calcification cluster in theplane of the breast. The remaining cluster was morphologi- reference frame of the MRI was potentially significant.cally the most pathologically indicative of DCIS. Biopsy of the calcification cluster proved positive for

The two mammogram views were registered to the DCIS.CE-MRI volume in the manner described previously and An area of significant debate in breast MRI is the extentthe location of the calcifications were colour coded and to which DCIS enhances with gadolinium DTPA. Somepresented to the radiologist in a 3D slice-based visualisa- radiologists suggest that there may be a subtle enhance-tion of the MRI volume. This is shown inFig. 24 where ment characteristic that can be used to detect DCIS. We arethe ‘trouble’ case of calcifications (‘3’) is visualised as a interested to pursue this as a larger study to see whether ared spherical region of interest (indicating the approximate definitive quantification of DCIS enhancement can beextent of the cluster). obtained. This could lead to the development of software

By providing a set of visual prompts to the radiologist tools that not only aid the localisation of mammogram-we had hoped to highlight regions that required additional visible microcalcifications in the MRI acquisition, but alsoscrutiny. Since one expects the contrast-enhancement provide analysis tools that help the radiologist localisecharacteristics of the calcifications to be very subtle in the DCIS-characteristic enhancement. In the interim, we useCE-MRI acquisition this provides a useful additional piece calcification localisation in conjunction with some simpleof information to the radiologist. Upon re-examination of segmentation techniques based on region-growing localthe MRI contrast sequence, the radiologist concluded that voxel contrast characteristics to produce visualizations ofthere was a region of slightly higher enhancement at the DCIS that we feel are useful to both the radiologist andindicated position of the third cluster, as indicated inFig. ultimately, the surgeon. An example of such a visualisation25. (based on a sagittal reslicing of the breast of interest) is

Although the enhancement characteristics shown in the shown inFig. 26.


Fig. 24. 3D visualisations of the location of the three calcification clusters in the reference frame of the MRI volume (labelled according to the previousfigure). This figure (and several other figures in this paper) is available in colour, see the on-line version.

9 .4. Case [3: multi-focal disease and correlated In terms of improving the bi-directional flow of in-pathological indicators formation between a mammographic and an MRI-based

examination, comparing multi-focality is a key issue.One of the difficulties with mammography-based assess- Multi-focal disease is more likely to be evident in MRI and

ment is the difficulty of determining the extent of multi- therefore if we wish to prospectively evaluate a mammog-focal disease. Sometimes a lesion visible in a mammogram ram, data fusion is useful. An example of this scenario isis only one of several present or in severe cases, bilateral shown inFig. 27,where a fairly evident mass highlightedlesions are not evident. With respect to MRI, one of the by calcifications is present in the mammogram and moremajor trade-offs is spatial resolution versus temporal than one region of focal enhancement is visible in thesampling of contrast uptake. Radiologist colleagues as- CE-MRI acquisition. This can be clearly seen by compar-sume that there is always a possibility of multi-focal ing the pre- and post-contrast MRI images.disease, particularly in patients who have exhibited recur- In the case of multi-focal disease, there is some benefitrences. Therefore despite the fact that improved temporal in being able to compare regions of enhancement in theresolution would give better (more discriminating) contrast MRI volume, with corresponding regions in the mammo-enhancement information, the clinical value of including gram. In this way, fine mammographic detail such asboth breasts in the examination is simply too important. calcifications and spiculations can be compared with focal


Fig. 25. (A,B) The location of calcification cluster ‘3’ in the corresponding coronal and transverse slices. Magnification of this area in the coronal slice for(C) pre- and (D) post-contrast indicates a slightly enhancing region.

enhancement in the MRI volume, enabling the radiologist rigid registration illustrates why radiologists appear toto evaluate pathology both in terms of the structural generally be unable to perform this comparison manually.characteristics for which mammography is very detailed, In fact, the radiologists we work with do not usually refercombined with the functional representation of disease in to the mammogram when assessing multi-focal conditionsCE-MRI. For the case shown inFig. 28, we illustrate a as it is considered possibly misleading. We believe that theselection of points in the MRI volume that were selected registration framework presented in this article is a stepby the clinician to compare the enhancement of suspicious towards making this comparison possible.regions with normal parenchyma contrast uptake. The As a final note to this usage scenario, we emphasise thatdisplayed points are actually selections across several CE-MRI provides a functional representation of the struc-slices displayed as a cumulative overlay to provide the tural detail in a mammogram. Mammographic assessmentclinician with a sense of the spatial distribution of points of large, poorly differentiated carcinomas, such produceswithin the reference plane of the current slice. The relatively little detailed information about the necrosis ofcorresponding points have been registered to the mammog- the tumour. Although it would be useful to be able to learnram views and are displayed in the lateral mammogram for something about the detailed pathological structure of thecomparison. The blue crosses indicate the two regions of tumour, this is often not possible with a mammogram. Onenhancement seen in the MRI volume. the other hand, the MRI contrast-enhancement characteris-

This is an interesting result in the sense that it illustrates tics provide valuable information about the vascularity andhow difficult it is for the radiologist to mentally perform necrosis of a tumour. The projected contrast enhancementthis comparison. The combination of landmarks selected from the MRI acquisition can produce a representation ofacross several MRI slices and the relative complexity of the pathology that can be compared against detail visible inthe internal deformation of the mammograms during non- the mammogram. As shown inFig. 29,this can be used to


Fig. 26. A rendering of the MRI volume (post-contrast sagittal acquisition) with the region corresponding to calcifications in the mammograms indicatedby arrows. In terms of understanding the pathology, 3D morphology is much more meaningful than a slice-based representation. Additionally, models suchas this may provide the surgeon with a useful location reference during excision.

Fig. 27. A suspicious region (indicated by calcification) in the lateral mammogram (left) compared with a pre- and post-contrast slice in the MRI. Theenhancement characteristics of the MR images suggest more than one enhancing region (circled).


Fig. 28. A collection of points in the lateral mammogram (A) corresponding to points selected across several slices in the MRI volume (displayed in B) asan overlay). The circled region in (A) encompasses the region inFig. 27, although it is clear from the MRI data that there is more than one enhancingregion (blue crosses).

Fig. 29. (A,F) The contrast enhancement projections corresponding to (C,H) in the mammogram (poorly differentiated lesions). In both cases, the MRIprojections illustrate small ‘cores’ of lower enhancement that typically correspond to necrosis. Surface rendering of these regions (B,G,E,I) provides aninteresting comparison of the shape of the two representations. (D) Simple thresholding and edge-detection of (C) provides a measure of substantiation thatthere is more than one component to the mass.


compare the spatial distribution of the necrosis of the MRI volume, or a biopsy/ localisation has been performed,tumour against the extent of the mass in the mammogram we compare the centroid of the backwards transformation(shown for two patient examples). from the MRI volume to the two mammographic views on

the basis of a lesion (or ROI) centroid. This analysiscentroid is then compared against the centroid of a manual

1 0. Registration results segmentation by the clinician. On the assumption thatdifferent classes of lesion will result in different deforma-

In the previous section, we illustrated through a series of tion results (and corresponding errors) we have groupedpatient cases, the clinical value of mammography to CE- each case into 7 species of pathology. These are shown inMRI data fusion. We have had the privilege of demon- Table 1.It is not terribly surprising that larger pathologiesstrating our software tools on a range of pro- and re- exhibit a more substantial registration error as the use oftrospective cases that we believe have made a contribution these regions as a salient feature for registration is moreto the patient management in several instances. As a matter ambiguous by virtue of our method for producing land-of course, the results of our registration and analysis can marks for deformation. Similarly, it is sometimes proble-only be accepted as a clinical possibility due to the matic to evaluate cases with calcifications as there may notdifficulty in validating non-rigid registration algorithms— be corresponding enhancement in the MRI volume for allparticularly between different modalities. In a few cases, clusters. Again, these considerations have limited the sizewe have had the benefit of a follow-up study involving of this study.image-guided localisation with either mammography or The set of results illustrated inTable 1 is somewhatMRI, although this would not be the usual case due to the difficult to compile as the deformations are calculated inpatient inconvenience and cost of such assessments. mm, but the voxel dimensions are not uniform across the

To date, our technique has been applied to 14 sets of various MRI acquisitions. Therefore, we have quoted thepatient data with good results. In two cases, we were error relative to a voxel as it better illustrates the fact thatunable to assess the registration due to the fact that in many of the cases, the MRI scans were acquired usingfeatures such as calcifications were not present in both transverse slicing and hence the voxel resolution in themammogram views. This represents a limitation of our MLO projection is poorer. This error trend is quite visiblemethod. In the remaining 12 cases (some of which were when the MLO voxel error is compared against the CCpresented in (Behrenbruch et al., 2000a,b), we have error. To further understand the relative significance of thisassessed the performance of this projection/ registration error, we compare the estimated error in mm (converted onapproach on a wide range of pathologies with very the basis of the average voxel dimension in both projec-different structural and tissue properties. To a certain tions) to the average calculated displacement field com-extent, the number of studies available for evaluation has puted from the registration transformation function (con-not been restricted by data availability, but rather clinical ceptually shown as a circular radius of deformation inFig.need for such a comparison (and consequentially, the30).clinical desire to spend time on follow-up analysis). In It can be seen from the chart inFig. 31, that althoughmost cases, women who have undergone an MRI scan the estimated registration error compared with the manualhave the necessary mammographic information available segmentation of the lesion is (in absolute terms) quiteto perform data fusion. A larger retrospective study could large, the displacement necessary to compute the trans-be performed to enable a measure of validation against formation from the compressed shape of the breast in theclinical cases where there is less ambiguity in the radiolo- mammogram to the prone shape of the MRI volume isgical assessment. substantial. On average across all seven classes of pathol-

Our evaluation of the performance of our registration, ogy, the registration error relative to the size of thealthough limited, is quite simple. In cases where pathology deformation is less than 20%. Although this is a prelimin-is visible in either /both of the mammogram views and the ary result, it provides an indication of the extent of the

T able 1A comparison of the registration error in 12 cases and categorised by 7 classes of pathology

Class Description CC Error MLO error Avg. error No. of cases

1 Small circular opacity 3.5 voxels 2.5 voxels 3.0 voxels 12 Spiculated opacity* 2.0 voxels 5.0 voxels 3.5 voxels 13 Infiltrating carcinoma 4.0 voxels 3.5 voxels 3.75 voxels 14 Fibroglandular Nodule 1.5 voxels 2.5 voxels 2.0 voxels 15 Large, dense circumscribed tumour 3.0 voxels 5.5 voxels 4.25 voxels 26 Multiple microcalcifications 2.5 voxels 3 voxels 2.75 voxels 57 Fibrous lesion (low enhancement) 2.5 voxels 6.5 voxels 4.5 voxels 1

*Associated with enlarged lymph node.


Fig. 30. An illustration of the local displacement field D calculated across the two successive registration steps. The local displacement field is calculated(from the transformation) for a point of interest and is used as a comparison with the registration error for a clinical ROI.

deformation necessary to achieve registration. Once again, when projected, underestimate the size of the region in thehowever, the extent of the local displacement of a region mammogram.varies between the two projective views but also dependsgreatly on the shape and size of the breast and to a certainextent, the slice orientation of the MRI acquisition.

In summary, although not trivial to validate, the maxi- 1 1. Discussionmum registration error we encountered was in the vicinityof 5–6 voxels, which in the case of a transverse MRI 1 1.1. General commentsacquisition (with 2–3-mm voxels in the MLO projection),this means a maximum error of typically 10 mm (depend- In this article we have endeavoured to present a noveling on the projection direction). Of course, in cases where concept in breast data fusion. To our knowledge, such anthe pathology is clearly visible in both modalities, this can approach has not yet been attempted in a clinical applica-itself be a registration landmark—hence in some instances, tion and therefore there are naturally a number of issuesvery good registration results were achieved with a per- that require further assessment. One of the major difficul-ceived accuracy of 2–3 mm. However, it must be consid- ties is validating a system that has a large number ofered that in these cases, the centroid error for larger lesions components and a relatively high degree of complexity.is also likely to be a somewhat inaccurate reference point There is little doubt that each component in the processas the extent of the visible lesion in the MRI volume will, involves a number of assumptions, and whilst we have

Fig. 31. A graph showing a comparison between the clinician evaluated registration error (with respect to features in the MRI projected back to the twomammogram views) and the local displacement resulting from the elastic deformation. On average, the registration error is less than 20% of the localdisplacement calculated from the transformation.


striven to minimise the heuristic nature of each step, there representative of studies that would require this kind ofis clearly room for improvement. fusion process. It is still, however, a consideration.

If such a system is to be clinically implemented and Additionally, our MRI projection depends on accurateconsidered useable by a radiologist, the key to success will characterisation of the tissue pharmacokinetics, an issueultimately lie in the software implementation and interface. that certainly warrants additional attention in order toThe techniques presented in this paper have been presented produce ‘pseudo mammograms’ that provide a mean-as automatic; however, in practice there is a need to be ingful basis for comparison with mammography.able to provide the clinician with interactivity at each step. • Motion correction. Although we have not discussed thisNot only is this likely to improve the clinician’s faith in issue extensively in this paper, MRI breast motionthe registration process, but it enables the ‘expert opinion’ correction is a critical issue. In a few cases where weto influence the quality of the result. To this end, our did not correct for motion (as the extent of thesoftware implementation of the techniques presented in pathology was clearly visible under enhancement) thethis paper (as a system) allows user input at each step of MRI projection was susceptible to cumulative motionthe process. artefacts. Whilst this may not detract from a point

In general terms there are a number of areas that need to comparison between views, it significantly affects thebe addressed as ‘caveats’ when implementing this kind of robustness of internal landmark detection.breast data fusion. Some of the obvious issues include: • Internal landmarks. Our wavelet-based feature detector• MRI resolution. Some of the data in our study used is a promising start towards compensating for the

slice-based 2D data (i.e. there are slice gaps), which internal deformation of the breast. We chose a waveletrequires volume interpolation prior to projection. The framework as it provides a large degree of flexibilityregistration error is strongly influenced by the voxel with respect to utilizing spatial filters that are mor-resolution and therefore this is clearly an issue. The phologically sensitive. In our case, the notion of agood news is that our current MRI protocol involves a feature is essentially a well-localised ‘bump’, howevertrue 3D acquisition, which improves the outcome there are many image characteristics such as curvilineardramatically. structures and the outline of the ductal disc that could

• MRI slice orientation. 3D MRI acquisition is usually be included in the registration framework to provide abased around a coronal slicing profile. Although a more accurate internal deformation.relatively high number of slices can be acquired (we • Deformation regularisation. The use of radial basishave examples with as high as 70 slices), this still limits functions for elastic image registration has proven to bethe projection resolution considerably. Ironically, the a useful, clinically intuitive and computationally effec-older transverse acquisitions that provided 2563256 tive way of performing the data fusion. By being able tovoxels in the CC projection plane produced better interact with the landmarks for registration and quicklyresults in some cases. In the future, if breast data fusion computing the registration result, the clinician is able tois required, it may be necessary to take several different efficiently achieve an acceptable fusion. However, weslice configurations in order to optimise the resolution accept that regularization, particularly in the approxi-of the different projection directions (i.e., a transverse mation context, is a complex issue and is currentlyand a sagittal view). somewhat heuristic. The notion of scale-controlled

• Breast compression. The key idea of this work is that smoothness has enabled us to partially reject thethe registration process ‘uncompresses’ the mammo- influence of small features that may introduce a regis-graphically imaged breast into the shape of the prone tration error; however, there is still much work to beMRI breast. Our model is simple and despite some done. Future efforts might involve the use of deforma-allowance for internal deformation via the scale-space tion models that pay greater heed to the biomechanicalfeature detection, there is room for improvement. properties of the tissues under deformation. Whilst thisPerhaps a combination of the physics-based models would not be a trivial exercise, considerable progress(Highnam and Brady, 1999)and a registration approach has been made with models that demonstrate soft tissuecould be the answer. Alternatively, there may be some characteristics that are quite realistic(Cotin et al., 1999;value in pursuing a registration process that involves Azar et al., 2000; Picinbono et al., 2000; Kyriacou etimaging a compressed breast with MRI. There are al., 1999).several interesting possible research directions here.

• Tissue characterisation. There are two issues here. 1 1.2. Future validationFirstly, if the breast is highly involuted in the mammo-gram views, then the internal deformation characteris- Mathematical and clinical validation of non-rigid regis-tics of the registration will be a poor approximation to tration is a problem that is an ongoing burden to thecompression compensation. Fortunately, in these cases computer vision community. We have provided a measurethe radio-opacity characteristics of the tissues in the of validation against deformation characteristics, clinicianmammogram are favourable, and in general are not segmentation and in a few cases we have had the benefit of


a localisation study. However, future research effort could transformations obtained via deformable models or regis-involve further validation exercises, including the follow- tration techniques. A few sample frames of such a se-ing: quence are shown inFig. 32, where a variation on a fast,

spoiled GRASS acquisition (20 frames/s) captures the1 1.2.1. Clinical population motion of a volunteer compressing the breasts in a

The case studies presented in this article have been transverse fashion. This example was not obtained with amotivated by the clinical need for fusion. However (at least breast coil, but with a cardiac surface coil above and belowin the UK) the majority of patients who present for a MRI the patient—however, it demonstrates the concept quiteexam have the necessary mammogram studies for fusion. well.It is conceivable that quite a large clinical study could beperformed on cases where the radiological ‘ground’ truth iswell understood. The problem with this may be the fact 1 2. Conclusionthat the lack of ambiguity in the clinical assessment isrelated to the characteristics of the mammograms (such as We have introduced a process of breast image datainvolution) and therefore in terms of compensation for fusion designed to aid the radiologist in understanding theinternal deformation of the breast, these studies may be of pathological relationship between mammography andlimited value. MRI, particularly when there may be ambiguity in the

diagnostic value of either modality. Our framework has1 1.2.2. Histology been developed to address a fairly wide range of imaging

The absolute ‘ground truth’ for validating non-rigid conditions with respect to MRI acquisition protocols andregistration is biopsy and wire localisation. We have had patient tissue characteristics. The results of the techniquesthe benefit of evaluating our technique with a few localisa- presented in this article have been assessed with a study oftion studies, however this would need to be performed on a 14 patients with promising performance in 12 of thesemuch wider scale. A possible protocol for this would be cases.MRI-guided wire localisation followed by a mammo- Our data fusion approach incorporates a combination ofgram(s) and core biopsy. MRI contrast-enhancement projection (to produce a ‘pseu-

do’ mammogram), boundary and internal landmark de-1 1.2.3. High-speed MRI tection and a two-stage registration process using radial

Earlier in our discussion we mentioned the possibility of basis functions. The internal deformation of the breast (duecompressing the breasts during the MRI acquisition to to mammography compression) is compensated by theprovide a better geometrical correspondence between MRI correlation of internal landmarks between the MRI projec-and mammography. Various kinds of MRI-imaged breast tion and the mammogram, using a novel wavelet-baseddeformation studies have been performed to look at tissue scale-space analysis. By combining boundary and internaldeformation characteristics from a diagnostic standpoint. landmark information, the registration approach is able toThere is a steady emergence of near real-time MRI cater for cases where there may or may not be internalsequences, particularly for applications such as cardiac landmarks due to the involution characteristics of themotion tracking. An interesting approach to validation of breast.breast registration would be to image the deformation of Finally, we presented an interesting set of clinical casesthe breast and treat the acquisition as a motion-tracking where our data fusion method was required to improveproblem. This could be compared against the kind of diagnosis or provide additional pathological information.

Fig. 32. Three frames taken from a high-speed MRI acquisition of breast deformation. In this simple example, the patient palpated her breast (handlocation illustrated by the arrows) whilst lying prone in the MRI machine between two cardiac surface coils. The internal deformation is quite clear andcould be used (in the context of a motion tracking approach) to evaluate deformation fields produced by non-rigid registration algorithms.


A zar, F.S., Metaxas, D.N., Schnall, M.D., 2000. A finite element model ofThese case studies were motivated by a clinical need tothe breast for predicting mechanical deformation during biopsyrectify the diagnostic disparity between mammography andprocedures. In: Proceedings of the IEEE Workshop on Mathematical

MRI, most often in cases where the patient exhibited dense Methods in Biomedical Image Analysis.breasts that did not image optimally with mammography. B ehrenbruch, C.P., Marias, K., Armitage, P.A., Brady, J.M., Clarke, J.,

Moore, N., 2000a. The generation of simulated mammograms fromWe then extended our set of case studies to a briefcontrast-enhanced MRI for surgical planning and postoperative assess-assessment of 12 (of 14) cases, which have been registeredment. In: Proceedings of IWDM (the International Workshop in

using our approach. Despite the difficulty with validating Digital Mammography), Toronto, Canada. Kluwer Academic Pub-non-rigid registration, particularly for multimodal applica- lishers, Dordrecht.

B ehrenbruch, C.P., Marias, K., Armitage, P.A., Yam, M., Moore, A.,tions, we have been able to demonstrate that the error dueEnglish, R.E., Brady, J.M., 2000b. Mammography-MRI 2D/3D datato registration (compared against a clinical evaluation) isfusion for breast pathology assessment. In: Proceedings of MICCAItypically less than 20% of the local displacement resulting(Medical Robotics, Imaging and Computer Assisted Intervention).

from elastic deformation. This is seen as an encouraging Springer, Pittsburgh, PA.result. B esl, P., McKay, N., 1992. A method for registration of 3-D shapes. IEEE

Trans. PAMI 14, 239–256.Our ongoing effort in this research is aimed at develop-B ookstein, F.L., 1989. Principal warps: ‘‘thin-plate splines and theing a more robust registration framework in the context of

decomposition of deformations’’. IEEE Transactions on Pattern Analy-a larger and (as indicated in the discussion) more diverse sis and Machine Intelligence 11 (6), 567–585.validation study. We have implemented the data fusion B ovik, A.C., Clark, M., Giesler, W.S., 1990. Multichannel texture analysis

using localized spatial filters. IEEE PAMI 12, 55–73.framework as a software package, which is currentlyB rix, G., Semmler, W., Schad, L.R., Layer, G., Lorentz, W., 1991.installed at the John Radcliffe Hospital, Oxford, where it is

Pharmacokinetic parameters in CNS Gd-DTPA enhanced MR imaging.used on a regular clinical basis. From our clinical ex- J. Comput. Assist. Tomogr. 15, 621–628.perience with our software tools, we have found that there B uckley, D.L., Kerslake, R.W., Blackband, J., Horsman, A., 1994.

Quantitative analysis of multi-slice Gd-DTPA enhancement dynamicis a considerable gulf between a theoretical fusion conceptMR images using an automated simplex minimisation procedure.with the implication of automatic performance, and aMagnetic Resonance in Medicine 32, 646–651.clinically useable (and accepted) system. Our long-term

B urt, P.J., Adelson, E.H., 1983. The Laplacian pyramid as a compactobjective is to produce a system capable of enabling the image code. IEEE Transactions on Communication 31 (4), 540–552.radiologist to do a routine comparison between mammo- C hristensen, G.E., Rabbitt, R.D., Miller, M.I., 1994. 3D brain mapping

using deformable neuroanatomy. Physics in Medicine and Biology 39,graphy and MRI609–618.

C oifman, R.R., Wickerhauser, M.V., 1992. Wavelets and adapted wave-form analysis. In: Benedetto, Frazier, (Eds.), Wavelets: Mathematicsand Applications. Studies in Advanced Mathematics. CRC Press, BocaA cknowledgementsRaton, FL.

C oifman, R.R., Meyer, Y., Wickerhauser, M.V., 1991. Adapted waveformThe authors would like to thank many people for their analysis, wavelet-packets and applications. In: Proceedings of

ˆ´contribution to this work. This includes Dr. Jerome ICIAM’91. SIAM Press, Philadelphia, PA, pp. 41–50.Declerck (Mirada Solutions) for useful discussions about C oifman, R.R., Meyer, Y., Quake, S.R., Wickerhauser, M.V., 1992. Signal

processing and compression with wavelet packets. In: Meyer, Y.,this work, Mr. Dermot Dobson and the staff at the OxfordRoques, S. (Eds.), Proceedings of the International ConferenceMRI Centre (John Radcliffe Hospital), Dr. Eve Coste-‘Wavelets and Applications’, Toulouse, France, 8–13 June, Obser-

` ´Maniere, Dr. Herve Delingette, Professor Nicholas Ayache ´ ´ ´vatoire Midi-Pyrenees de l’Universite Paul Sebatier, pp. 77–93.and Dr. Alexis Roche (Mirada Solutions Ltd.) for useful C otin, S., Delingette, H., Ayache, N., 1999. Real-time elastic deformationfeedback and discussion. The idea of using MRI to of soft tissues for surgery simulation. IEEE Transactions on Visualiza-

tion and Computer Graphics 5 (1), 62–73.potentially validate breast tissue deformation was theD ’Astous, F., Jernigan, M.E., 1984. Texture discrimination based oninspiration of Professor Elliot McVeighe (Johns Hopkins/

detailed measures of the power spectrum. In: Proceedings of the 7thNIH Bethesda) as well as the preliminary breast data. International Conference on Pattern Recognition, pp. 83–86.Finally, C. Behrenbruch would like to gratefully acknowl- D aubechies, I., 1992. In: Ten Lectures on Wavelets. CBMS-NSF Regionaledge the Commonwealth Scholarship and Fellowship Conference Series in Applied Mathematics, Vol. 61. SIAM Press,

Philadelphia, PA.Program for his doctoral funding at Oxford University. P.D aubechies, I., Grossman, A., Meyer, Y., 1986. Painless nonorthogonalArmitage and J.M. Brady acknowledge the EPSRC for

expansions. Journal of Mathematical Physics 27, 1271–1283.funding (GR/M54995). D eclerck J., Feldmar J., Goris M.L., Betting F., 1996. Automatic

registration and alignment on a template of cardiac stress and restSPECT images. Mathematical Methods in Biomedical Image Analysis.

E vans, A.C., Dai, W., Collins, L., Neelin, P., Marrett, S., 1991. Warping ofR eferences a computerized 3-D atlas to match brain image volumes for quantita-

tive neuroanatomical and functional analysis. In: Leow, X. (Ed.), Proc.A rad, N., Reisfeld, D., 1995. Image warping using few anchor points and SPIE 1445, Medical Imaging V: Image Processing. San Jose, CA,

radial functions. Computer Graphics Forum 14 (1), 35–36. USA, pp. 236–246.A rmitage, P., Behrenbruch, C.P., Brady, M., 2001. A fast, reliable method F letcher, S.W., Black, W., Harris, R., Riner, B.K., Shapiro, S., 1993.

for calculated T in 3D contrast-enhanced breast MR imaging. In: Report of the International Workshop on Screening for Breast Cancer.1

Proc. ISMRM’01, Glasgow. J. Natl. Cancer Inst. 85, 1644–1656.


G ilhuijs, K.G.A., 1995. In: Automated Verification of Radiation Treat- Human Representation and Processing of Visual Information. W.H.ment Geometry. Universiteit van Amsterdam, Febo-druk, Enschede- Freeman, San Francisco, CA.Utrecht, The Netherlands. N ystrom, L., Rutqvist, L.E., Wall, S., Lindgren, A., Lindqvist, M., Ryden,

G rossman, A., Morlet, J., 1985. Decomposition of functions into wavelets S., Andersson, I., Bjurstam, N., Fagerberg, G., Frisell, J., Tabar, L.,of constant shape and related transforms. In: Streit, L. (Ed.), Mathe- Larsson, L.G., 1993. Breast cancer screening with mammography:matics and Physics, Lectures on Recent Results. World Scientific, overview of Swedish randomised trials. Lancet 341, 973–978.Singapore. P icinbono, G., Delingette, H., Ayache, N., 2000. Real-time large displace-

H aralick, R.M., Shanmugam, K., Dinstein, I., 1973. Texture features for ment elasticity for surgery simulation: non-linear tensor-mass model.image classification. IEEE SMC 3 (6), 610–621. In: Proceedings of MICCAI (Medical Robotics, Imaging and Com-

H ardy, R.L., 1981. Multiquadric equations of topography and other puter Assisted Intervention), Pittsburgh, USA. Springer, Berlin.irregular surfaces. Journal of Geophysical Research 76 (8), 1905– P incus, S.M., 1992. Approximate entropy: statistical properties and1915. applications. Commun. Statist. Theory Methods 21, 3061–3077.

H ayton, P., Brady, M., Tarassenko, L., Moore, N., 1997. Analysis of P incus, S.M., 1997. Not all (possibly) ‘random’ sequences are createddynamic MR breast images using a model of contrast enhancement. equal. Proc. Natl. Acad. Sci. USA 94, 3513–3518.Medical Image Analysis 1 (3), 207–224. R ankin, S.C., 2000. MRI of the breast (Review Article). Br. J. Radiol. 73,

H ighnam, R.P., Brady, J.M., 1999. In: Mammographic Image Processing. 806–818.Kluwer Academic Press, Dordrecht. R euckert, D., Sonoda, L.I., Hayes, C., Hill, D.L., Leach, M.O., Hawkes,

H olschneider, M., 1995. In: Wavelets: An Analysis Tool. Oxford Mathe- D.J., 1999. Nonrigid registration using free-form deformations: appli-matical Monographs. Oxford University Press, London. cation to breast MR images. IEEE Transactions in Medical Imaging 18

H orn, B.K.P., Schunck, B.G., 1981. Determining optical flow. Artificial (8), 12–21.Intelligence 17, 185–203. R ohr, K., Stiehl, H.S., Sprengel, R., Beil, W., Buzug, T.M., Weese, J.,

K arssemeijer, N., Thijssen, M., Hendriks, J., van Erning, L. (Eds.), 1998. Kuhn, M.H., 1996. Point-based elastic registration of medical imageDigital Mammography, Computational Imaging and Vision. Kluwer data using approximating thin-plate splines. Lecture Notes in Com-Academic Publishers, Dordrecht. puter Science 1131, 297–306, Springer.

¨K avanagh, A.M., Mitchell, H., Giles, G.G., 2000. Hormone replacement R uprecht, D., Muller, H., 1993. Free form deformation with scattered datatherapy and accuracy of mammographic screening. Lancet 355, 270– interpolation methods. Computing Supplementum 8, 267–281.274. S allam, M.Y., Bowyer, K.W., 1996. Registration and difference analysis of

K ita, Y., Highnam, R., Brady, M., 1998. Correspondence between corresponding mammogram images. Medical Image Analysis 1 (1),different view breast X-rays using a simulation of breast deformation. 73–91.In: IEEE Computer Society Conference on Computer Vision and S tein, E., 1979. In: Singular Integrals and the Differentiability Property ofPattern Recognition, pp. 700–707. Functions. Princeton University Press, Princeton NJ.

K oenderink, J.J., 1984. The structure of images. Biological Cybernetics S weldens, W., 1996. The lifting scheme: a custom-design construction of50, 363–370. biorthogonal wavelets. Appl. Comput. Harmon. Anal. 3 (2), 186–200.

K ovesi, P., 1996. Invariant measures of imag4e features from phase T hirion, J.-P., 1998. Image matching as a diffusion process: an analogyinformation. Ph.D. thesis, Department of Psychology, University of with Maxwell’s demons. Medical Image Analysis 2 (3), 243–260.Western Australia. T ofts, P.A., Kermode, A., 1991. Measurement of the blood–brain barrier

K yriacou, S.K., Davatzikos, C., Zinreich, S.J., Bryan, R.N., 1999. and leakage space using dynamic MR imaging 1. FundamentalNonlinear elastic registration of brain images with tumour pathology concepts. Magnetic Resonance in Medicine 17, 347–367.using a biomechanical model. IEEE Trans Med. Imag. 18 (7), 580– T urner, M.R., 1986. Texture discrimination by Gabor functions. Bio-592. logical Cybernetics 55, 71–82.

L anyi, M., 1996. In: Diagnosis and Differential Diagnosis of Breast V erbeek, A.L.M., Hendricks, J.H.C.L., Holland, R., Mravunac, M.,Calcifications. Springer, Berlin. Strumans, F., Day, N.E., 1984. Reduction in breast cancer mortality

L eung, W., Goldberg, F., Zee, B., Sterns, E., 1997. Mammographic through mass screening with modern mammography. (First results ofdensity in women on postmenopausal hormone replacement therapy. the Nijmegen Project 1975–1981). Lancet i, 1222–1224.Surgery 122 (4), 669–673 (discussion 673–674). W arren, R.M.L., Hayes, C., Advisory Group of the UK Study of MRI

L itherland, J.C., Evans, A.J., Wilson, A.R., 1997. The effect of hormone Screening for Breast Cancer – MARIBS, 2000. Localization of breastreplacement therapy on recall rate in the National Health Service lesions shown only on MRI – a review for the UK study of MRIBreast Screening Programme. Clin Radiol 52 (4), 276–279. screening for breast cancer. Br. J. Radiology 73, 123–132.

L ittle, J.A., Hill, D.L.G., Hawkes, D.J., 1997. Deformations incorporating W ickerhauser, M.V., 1994. In: Peters, A.K. (Ed.), Adapted Waveletrigid structures. Computer Vision and Image Understanding 66 (2), Analysis from Theory to Software. IEEE Press.223–232. W irth, M.A., Choi, C., 2000. The comparative analysis of mammograms:

M allat, S., 1998. In: A Wavelet Tour of Signal Processing. Academic a nonrigid approach to registration. In: Proceedings of IWDM (Inter-Press, San Diego, CA. national Workshop in Digital Mammography), Toronto, Canada.

M allat, S.G., 1989. Multiresolution approximation and wavelet orthonor- Elsevier, Amsterdam.2mal basis of L (R). Transactions of the AMS 315, 69–87. W itkin, A.P., 1983. Scale-space filtering. In: 8th International Joint

M arias, K., Brady, J.M., Parbhoo, S., Seifalian, A.M., 1999. Registration Conference on Artificial Intelligence. Karlsruhe, Germany.and matching of temporal mammograms for detecting abnormalities. W olfe, J.N., 1982. In: Xenoradiography of the Breast. Thomas, Spring-In: Proceedings of Medical Imaging Understanding and Analysis field, IL.(MIUA’99). Oxford, UK. Y am, M., Brady, J.M., Highnam, R.P., English, R.E., Kita, Y., 1999.

M arias, K., Behrenbruch, C.P., Brady, J.M., Parbhoo, S., Seifalian, A.M., Reconstructing microcalcification clusters in 3-D using a parame-2000. Multi-scale landmark selection for improved registration of terized breast compression model. In: Computer Assisted Radiologytemporal mammograms. In: Proceedings of IWDM (International and Surgery (CARS), Paris.Workshop in Digital Mammography), Toronto, Canada. Elsevier, Z hu, S.C., Yuille, A., 1996. Region competition: unifying snakes, regionAmsterdam. growing, and Bayes/MDL for multiband image segmentation. IEEE

M arr, D., 1982. In: Vision: A Computational Investigation Into the Trans. PAMI 18 (9), 884–900.

F usion of contrast-enhanced breast MR and mammographic ...kmarias/4_CPB_MEDIA.pdf · Christian P....

Documents

Transcript of F usion of contrast-enhanced breast MR and mammographic ...kmarias/4_CPB_MEDIA.pdf · Christian P....