Detection of Filamentous Structures in Low-Contrast

Images Acquired in Defocus Pair by

Cryo-Electron Microscopy

Yuanxin Zhu, Bridget Carragher,

David Kriegman and Clint S. Potter

(Submitted to IEEE Computer

Society Conf. Computer Vision and Pattern

Recognition, Hawaii)

Date Issued: June 2001

The Beckman Institute Imaging Technology Group

Technical Report 01-010

Copyright © 2001 Board of Trustees of the University of Illinois

The Beckman Institute for Advanced Science and Technology

Imaging Technology Group 405 N Mathews

Urbana, IL 61801 techreports@itg.uiuc.edu http://www.itg.uiuc.edu

Detection of Filamentous Structures in Low-Contrast ImagesAcquired in Defocus Pair by Cryo-Electron Microscopy

Yuanxin Zhu, Bridget Carragher, #David Kriegman, and Clinton S. Potter

Beckman Institute, and #Department of Computer Science, University of Illinois, Urbana, Illinois 61801

AbstractSince the foundation of the three-dimensional image

reconstruction of helical objects from electronmicrographs was laid more than 30 years ago, there havebeen sustained developments in specimen preparation,data acquisition, image analysis, and interpretation ofresults. However, the boxing of filaments in large numberof images—one of the critical step toward thereconstruction in atomic resolution—is still restrained bymanual processing even though interactive interfaces havebeen built to aid the tedious and less accurate manualboxing. This article describes an accurate approach forautomatically detecting filamentous structures in low-contrast images acquired in defocus pair using cryo-electron microscopy. The performance of the proposedapproach has been evaluated across variousmagnifications and a series of defocus values using as atest driver specimens of tobacco mosaic virus (TMV)preserved in vitreous ice. By integrating the proposedapproach into our automated data acquisition andreconstruction, we are able to get a 3D map of TMV at theresolution of 10Å within hours in a fully automatic way.

1. Introduction

The next decade promises to be an exciting one forunderstanding the inner workings of the cell. Genomes arebeing sequenced at an ever-increasing pace, and proteinexpression systems are being perfected, so we can expectthat many of the polypeptide building blocks of the cellwill be available for detailed study soon. It is clear that themajority of a cell’s proteins do not function alone but arerather organized into macromolecular complexes. Thesecomplexes can be thought of as machines containingfunctional modules (the enzymatic core), regulatorymodules (controlling the core), and localization modules(directing the complex to a particular part of the cell).Examples include microtubule-motor complexes,actomyosin, nuclear pore complexes, nucleic acidreplication and transcription complexes, vaults, ribosomes,proteosomes, etc.

Cryo-electron microscopy (Cryo-EM) is an approachthat can be used to provide valuable structural informationon such complexes. Briefly, the technique uses atransmission electron microscope (TEM) to acquire 2D

projection images of a specimen preserved in vitreous ice,allowing the specimen to be examined in its native state[1]. A 3D electron density map can be reconstructed fromthe 2D projections using what are essentially tomographicreconstruction techniques. The particular technique isdictated by the symmetry and geometry of the specimen.

Cryo-EM as a technique does however suffer from oneserious drawback – it is very time consuming and much ofthe work requires manual data processing and is extremelytedious. One of the principal bottlenecks to this techniqueis the enormous number of images that must be acquiredfor the structural analysis. This requirement results fromthe necessity for using extremely low doses of electrons(10e/A^2) to image the specimen in order to avoid beaminduced radiation damage. As a result the acquired imageshave a very low signal to noise ratio, and in order toreconstruct a 3D electron density map a very large numberof images must be averaged together. The exact number ofimages required to complete a 3D reconstruction of amacromolecule depends on its size and on the desiredresolution. It is generally agreed that in order to interpret astructure to atomic resolution, data from possibly hundredsof thousands of copies of the macromolecule must be used[2]. This in turn requires the collection of thousands totens of thousands of electron micrographs. One of thecurrent constraints in the field is the manual dataacquisition methods that are slow and labor-intensive, andresult in sometimes low percentages of suitable images.This is where we have demonstrated that techniques fromcomputer vision have proven to be extremely effective andare opening the door to rapid advances in experimentalstructural biology. The paper describes how thesetechniques are used to solve a specific aspect, and in turninforms the computer vision and pattern recognitioncommunity of new opportunities for applying visionmethods within a new domain.

One could compare the state of molecular microscopytoday to the situation in the x-ray crystallography fieldsome 20 years ago. Since then the x-ray crystallographershave expended a great deal of effort on automating thetime consuming and tedious aspects of their work. Theresult is that today x-ray structure determinations can bedone fast and efficiently – as evidenced by the veritableexplosion of new structures appearing in the journals.Cryo-EM is ready for a similar developmental effort.

Using helical objects as the driver, we have beendeveloping an integrated system for cryo-EM that wouldallow a 3D electron density map to be automaticallygenerated within hours of a specimen being inserted in themicroscope with no manual intervention. This wouldvastly improve both the quality and quantity of datacollection and processing. Towards this goal, we havealready automated acquiring images from the electronmicroscope [3,4]. The system performs all the operationsnormally performed by an experienced microscopistincluding multi-scale image analysis to identify suitabletargets for high magnification image acquisition. Thesystem can acquire up to 1000 images a day and theoverall performance is equivalent to that of an experiencedmicroscopist.

More than 30 years ago, DeRosier and Moore [5] laidthe foundation for the 3D reconstruction of biologicalstructures with helical symmetry from electronmicrographs. Since then, tools have been developed to aidthe image processing of helical objects [6-15]. Althoughinteractive interfaces (for example, the IMGBOX [15])have been built for extracting (boxing out regions of)filaments in images where multiple filaments at variousorientation and length may present, they are essentiallymanual limited both in speed and accuracy. Integration ofthe automated data acquisition with the 3D reconstructionprocess requires an automatic boxing process, i.e. foridentifying helical filaments in projection images. Thisautomation is particularly promising since the boxing stepis often the bottleneck for high-resolution reconstruction[16]. To our best knowledge, no such technique has beenreported so far.

This paper presents an accurate approach forautomatically detecting filamentous structures in low-dose, low-contrast images acquired in defocus pair usingcryo-EM; the first image is acquired at very near to focus(NTF) condition (e.g., -0.3 ~ -0.15µm, “-” and “+”indicate under-focus and over-focus respectively) and thesecond one at a farther from focus (FFF) (e.g., -3 ~ -2µm), see Fig. 2, for example. The time interval betweenthe two exposures is less than 1s. There are at least twomajor advantages of using a defocus pair of images. First,by combining the two images in the defocus pair,relatively high contrast at both low and high spatialfrequencies can be attained. Second, the moderatelystrong low-resolution signals in the FFF images make itpossible for us to develop algorithms to identifyfilamentous structures automatically. The idea of using adefocus pair of images has been explored by several otherresearchers [17-20].

Our approach begins with a three-level perceptualorganization algorithm to detect filaments in the FFFimages, see Fig. 1 for a schematic overview. Because theNTF images are acquired very close to focus (e.g. -0.15µm), the contrast in such images is extremely low. On

the other hand, the NTF image in a defocus pair covers thesame specimen area as the FFF image. The relativedistance between filaments within the NTF image shouldbe the same as that of the FFF image. We thereforecircumvent directly spotting filaments in the NTF imageand instead detect filaments in the FFF image which isthen aligned using phase correlation techniques with theNTF image. After alignment, the filaments are extractedfrom the NTF image using the coordinates identified in theFFF image.

Det

ectio

nof

fila

men

tsin

the

FFF

imag

eus

ing

ath

ree-

leve

lpe

rcep

tual

orga

niza

tion

Edge detection

Discontinuous edgesgrouped into line segments

Line segments organizedinto filaments

High magnification images in defocus pair; avery near-to-focus (NTF) image is acquired

first, followed by a far-from-focus (FFF)image from any chosen specimen area.

Extraction of filaments in NTF image usingcoordinates as identified in the FFF image

Alignment of the NTF image to the FFF image

Figure 1. Schematic overview of the automatedfilament finding approach.

The performance of the proposed approach was testedand evaluated by applying it to high magnificationmicrographs of tobacco mosaic virus preserved in vitreousice. Performance is assessed in terms of the percentagesof correctly identified filaments, false alarms (incorrectlyidentified areas), and false dismissals (unboxed filaments).The calculation is based on a comparison to the filamentsthat would have been selected by an expert. Falsedismissals do not constitute a serious type of error as longas there are many filament images available and thepercentage of falsely dismissing is low. False alarms do

pose a severe error since they are not projections of realhelical filaments and thus introduce significant error intothe reconstruction process. Therefore, a preferred filamentdetection approach should produce a high percentage ofcorrectly identified filaments while maintaining a very lowpercentage of false alarms.

2 Materials and Methods

2.1 Image acquisitionOur automated acquisition system [3, 4] was used to

record high magnification images of helical filaments oftobacco mosaic virus (TMV) preserved in vitreous ice.TMV is a well-characterized helical virus [21]. It is oftenused as a transmission electron microscopy (TEM)standard for calibration of magnification and determiningresolution and provides an ideal test specimen forevaluating a new technique. Defocus pairs of images werecollected using a Philips CM200 TEM quipped with a1Kx1K CCD camera with phospor scintillator. At thismagnification the pixel size was 2.8Å and the accumulateddose for any chosen specimen area was about 10e/A^2.An example of a defocus pair of images acquired in thisway is shown in Fig. 2(a) and (b). The images showsprojections of TMVs. These structures form helicalfilaments 18 nm in diameter and 300 nm in length.

(a) (b)

(c) (d)

Figure 2. Illustration of filament finding in adefocus pair of images of specimens of tobaccomosaic virus (TMV) (6,6000x) acquired usingcryo-electron microscopy (EM). The imageshown in (a) was acquired first at a very near-to-

focus (NTF) condition (-0.3µm) and the oneshown in (b) was recorded second at muchfarther from focus (FFF) (-3µm). Followingfilament finding in the FFF image, alignment ofthe NTF image to the FFF image results in 7filaments found in the pair of images shown in (c)and (d) respectively where targeted filamentsoutlined by pairs of parallel line segments.

2.2 Introduction to perceptual organizationIn humans, perceptual organization is the ability to

immediately detect relationships such as collinearity,parallelism, connectivity, and repetitive patterns amongimage elements [22]. Researchers in human visualperception, especially the Gestalt psychologists [23,24],have long recognized the importance of findingorganization in sensory data. The Gestalt law oforganization states the common rules by which our visualsystem attempts to group information [23,24]. Some ofthese rules include proximity, similarity, common fate,continuation and closure. In computer vision, as Mohanand Nevatia [25] proposed, perceptual organization takesprimitive image elements and generates representations offeature groupings that encode the structuralinterrelationships between the component elements. Manyperceptual grouping algorithms [25-28] have beenproposed, and surveys of these works can be found [22,29-31].

2.3 Detecting filaments in FFF imagesWe will model the filament detection process in the

FFF images as a three-level perceptual organization basedon the terminology proposed by Sarkar and Boyer [30]who arrange the features to be organized into fourcategories: signal, primitive, structural, and assembly. Ateach level, feature elements that are not grouped intohigher-level features are discarded. Briefly, at the signallevel, a Canny edge detector [32] is used to detect weakfilament boundaries. A Hough transform [33] followed bydetecting end points of line segments and mergingcollinear line segments is developed at the primitive levelto organize discontinuous edges into line segments with acomplete description. At the structural level, linesegments are grouped into filamentous structures, areasenclosed by pairs of straight lines, by seeking parallelismand utilizing high-level knowledge. In addition, statisticalevidence is used to separate two filaments if they arejoined together end to end.

Edge detection For noisy images, it is necessary that atthe signal level, edge detection be used to enhance low-level feature elements by organizing interesting boundarypoints into edge chains and suppressing the possibleeffects of noise on higher-level perceptual organization.Edge detection is typically a three-step process including

noise smoothing, edge enhancement and edge localization[34]. We adopt probably the most widely used edgedetector in today’s machine vision community, the Cannyedge detector, to detect these weak boundaries. Selectingthe input parameters of the Canny algorithm is a criticalstep because the resulting edge quality varies greatly withthe choice of parameters. At the signal level, the inputparameters were selected to optimize the quality of edgesfor the purpose of higher-level perceptual organization.The best single overall parameter set for the Canny edgedetector, identified by the performance measure methodand evaluated on a wide variety of real images [35], doesnot produce a successful result as applied to images in ourcase, see Fig. 3, for example. To select the inputparameters suitable for our highly noisy, low-contrastcryo-EM images, we first select a range of parameters thatsamples the space broadly enough but not too coarsely foreach parameter, and then chose the best suitable inputparameter by visual inspection of the resulting edgeimages. While this selection does not guarantee thatoptimal input parameter set was identified, it doesgenerate good results in a timely way. Fig. 3 (a) and (b)show the edges detected using different parameter set.

Line detection and end points computing At theprimitive level, the Hough transform (HT), followed bysearching end points of the line segments and mergingcollinear line segments, is used to organize noisy anddiscontinuous edges into complete line segments. Tosimplify the computation, the θρ − rather than slope-

intercept parameters are used to parameterize lines [36].The use of the HT to detect lines in an image involves thecomputation of HT for each pixel in the image,accumulating votes in a two dimensional accumulatorarray and searching the array for peaks holdinginformation of potential lines present in the image. Theaccumulator array becomes complicated when the imagecontains multiple lines, with multiple peaks, some ofwhich may not correspond to lines but are artifacts ofnoise in the image. Therefore, we iteratively detect thedominant line in the image, remove its contribution fromthe accumulator array, and then repeat to find thedominant of the remaining lines.

The peaks in the accumulator array provide only theshortest distance of the line to origin ( ρ ) and the angle

that the normal makes with the x-axis (θ ) of the imageplane. They fail to provide any information regarding thelength, position, or end points of the line segments, whichare vital to the detection of acceptable filaments. Severalalgorithms have been proposed in the literature for findingthe length and end points of a line from the Houghaccumulator array [37-39]. The drawback of thesealgorithms is that they are computationally intensive. Ourintegrated system requires that the computation be

performed on the order of seconds to maximizethroughput.

We extend an algorithm, due to McLaughlin and Alder[40], for computing the end points of a line segmentdirectly from the image, based on accurate line parametersdetected by HT. Accurate line parameters can be detectedby using HT since most noise has been removed by theedge detection process. At each iteration, we find theglobal maximum in the accumulator array and from thiscompute the equation (parameters) of the correspondingline. A simple post-processing step is used to find thebeginning and end points of the line, which involvesmoving a small window (e.g. 1515 × pixels) along the lineand counting the number of edge pixels in the window.This count will rise above a threshold value at thebeginning of the line and decrease below the threshold atthe line end. Collinear line segments are merged byallowing gaps in the line segment while moving the smallwindow along the line.

Having found two end points of the line segment, wenext tag all pixels lying on the line segment and removethe contribution of these pixels to the Hough accumulatorarray. The Hough accumulator array is now becomingequivalent to that of an image that had not contained thedetected line segment. The above processing is repeatedwith the new global maximum of the accumulator array.This continues until the maximum is below a thresholdvalue which, found from experimentation, is recalculatedat each iteration. Edges that are not grouped into linesegments are discarded after primitive-level perceptualorganization. Fig. 3(c) shows an example of the linesegments with acceptable length obtained by primitive-level perceptual organization where we can see that theend points of those line segments were accuratelyidentified and collinear line segments were successfullymerged.

Detection of filamentous structures At the structurallevel, line segments are actively grouped into filaments byseeking parallelism and employing knowledge obtainedfrom training data. A filament is the area enclosed by twoline segments with a particular structural relationship.Besides parallelism, the two line segments should alsomeet some heuristics: (i) The distance between the twoline segments should be between specified maximal andminimal values, both of which are determined bystatistical analysis on training data. (ii) The shorter one ofthe two line segments should be no less than one third ofthe length of the longer line segment. (iii) There should besome overlapped area between the two line segments. Theoverlapped area can be measured from the correspondencebetween the end points of the two line segments. At leastone third of either one should overlap with the other linesegment. The area enclosed by two line segments thatmeet these requirements is considered as a possiblefilament. Finding every possible filament requires that the

algorithm search for all possible matches in aneighborhood of each line segment. As a result, there maybe situations where a line segment has multiple matches(filaments). In this case, the match with the largestoverlapped area wins the competition. Therefore, each linesegment can be a boundary of only one filament. Like theungrouped edges in the primitive-level, unmatched linesegments are discarded.

(a) (b)

(c) (d)

Figure 3. Illustration of filament finding in the FFFimage of a defocus pair using a three-levelperceptual organization algorithm. (a) and (b)Edges detected in the image, shown in Fig. 2(b),using the parameter set identified by Heath et al.[35] and that we selected, respectively. (c) Linesegments with acceptable length obtained by theprimitive-level perceptual organization. (d) 7filaments were detected after structural-levelorganization.

In selecting filaments from the images that will bepassed to the 3D reconstruction algorithms, more stringentconstraints must be used to limit the number of filamentsin order for the reconstruction process to be accurate andefficient. First, only the area enclosed by the overlappedpart of the two line segments are clipped as the detectedfilament. Secondly, as described earlier, filaments may bejoined end to end along their length (see Fig. 4(a), forexample) and must be individually segmented prior to the3D image reconstruction. Thus our algorithm mustseparate a filament aggregation into multiple segments if itconsists of two or more filaments that are joined together

end to end (the longest filament in the image shown in Fig.3(e), fox example). Finally, filaments that are shorter thanthe minimal length required for further analysis arediscarded.

Separation of end-to-end joins To detect whether afilament actually consists of two or more shorter filamentsthat are end-to-end joined together, we have adopted afour-step approach. (i) Rotate the image around its centerso that the inclined filament becomes horizontal, and thencut the filament out of the rotated image. (ii) Enhancepossible end-to-end joins by linearly filtering the filamentimage with a Gaussian kernel and computing the gradientmagnitude for each pixel in the smoothed image. (iii)Sum along the columns of the image to generate a 1Dprojection in which the global peak corresponds to theposition of a possible end-to-end join. (iv) Determine thelocation of the end-to-end joins by thresholding based onthe statistics (maxium, mean, and standard deviation) ofthe 1D projection. Using this method the end-to-end joinwas identified in Fig. 4(a) and the longest filament asshown in Fig. 4(b) is extracted for further analysis.

After examining a set of filament images, we foundthat the metric defined as ( ) Wm µ− can be used to

accurately discriminate whether a filament contains end-to-end joins, where m and µ are, respectively, the

maximum and the mean of the 1D projection, and W is thewidth of the filament. First, a threshold value is learnedfrom the set of images by visual inspection. Then afilament is classified as one containing one or more end-to-end joins if the value of the metric is larger than thethreshold value and the filament will be separated alongthe position indicated by the global peak in the 1Dprojection. A recursive approach is used when there aremultiple end-to-end joins within one filament. In otherwords, a filament is split into two sub-filaments along itsdominant split point indicated by the global peak in the 1Dprojection. The sub-filaments are examined using thesame approach, split them if there are end-to-end joins,repeat this process until all filaments do not contain anyend-to-end join.

(a)

(b)

Figure 4. Illustration of end-to-end separation. (a)An extracted region from the image, shown inFig. 2(d), represents two filaments aggregatedend to end as indicated by the arrow. (b) Thelongest filament in (d) after separation along theend-to-end join.

2.4 Detecting filaments in NTF imagesAs mentioned earlier, one of the advantages of image

acquisition in defocus pair is to use information aboutfilaments in the FFF image of the defocus pair to aidfilament finding in the NTF image of the pair. We firstalign the NTF images to their corresponding FFF imagesthen extract filaments in the NTF images using thecoordinates as identified in the corresponding FFF images.An example of a pair of NTF and FFF images with 7filaments successfully targeted in each image is shown inFig. 2.

Cross-correlation has been the most widely exploitedtool for aligning pairs of images acquired under differentconditions [41]. The cross-correlation function (CCF) isusually calculated by first forming the cross powerspectrum (the product of multiplying the complexconjugate of the Fourier transform of the first image by thetransform of the second) and then inverse-transforming thecross power spectrum back to real space. Ideally, the CCFexhibits a well-posed local peak, the position of whichindicates the relative displacement of the two images.However, as [42] observed, the cross-correlation peakdeteriorates rapidly with increasing defocus differencebetween a pair of images. Our initial experimentconfirmed that the peak shape of the CCF between a pairof NTF and FFF images is generally unrecognizable (seeFig. 5 (a)).

(a) (b)

Figure 5. The phase correlation and cross-correlation surfaces obtained from the defocuspairs of images shown in Fig. 2(a) and (b). Thepeak (indication of relative displacement of thetwo images) on the phase-correlation surface ismuch sharper than that on the cross-correlationsurface.

In order to improve the peak shape of the CCF undernoise or strong background variations, several variantsinvolving linear or non-linear modification in Fourierspace has been previously proposed (for review, see [43]).Particularly, Saxton [43] proposed two modifications foraccurately aligning sets of images. The first modification,called a phase-compensated CCF, was supposed toaddress general case, when the transfer functions withwhich the images have been recorded are arbitrarily

complex, but requires at least an approximate knowledgeof the transfer functions in force. The secondmodification, proposed particularly for axial imaging withcontrast dominated either by phase or amplitude as is truein our case, does not require any knowledge of the transferfunctions, called a phase-doubled CCF [43]. The phase-doubled CCF is calculated as the inverse Fourier transformof the phase difference between two images (the crosspower spectrum of the two images divided by itsmodulus). We noticed interestingly that the phase-doubled CCF is essentially the same as the phasecorrelation alignment method developed by Kuglin andHines [44]. As indicated by the latter, the phasecorrelation is a highly accurate alignment technique thatexhibits an extremely narrow correlation peak (see Fig.4(b), for example) and is generally insensitive to narrowbandwidth noise and conventional image degradations.

We therefore adopt the phase correlation technique toalign pairs of defocus images. In summary, the algorithmfor aligning a defocus pair of images using the phasecorrelation technique consists of four steps: (i) A 2D fastFourier transform is computed for the NTF and FFFimages which have the same dimensions, resulting in twocomplex arrays. (ii) The phase difference matrix isderived by forming the cross power spectrum and dividingby its modulus. (iii) The phase correlation function (PCF)is then obtained as a real array by taking the inverse FFTof the phase difference matrix. (iv) The relativedisplacement of the two images is finally determined bysearching the position of the highest peak in the PCF. Ingeneral, the position of PCF surface peak is a continuousfunction of image displacement. Since the PCF surfacepeak is very sharp, it should be straightforward to measuresubpixel (non-integer) displacements through the use ofinterpolation with a few data points around the peak. Inour case, it is accurate enough to align a defocus pair ofimages by finding an integer displacement between thetwo images (see Fig. 2(c), for example).

3. Experiment result and analysis

We have tested and evaluated the performance of theproposed approach by applying it to high magnificationimages of helical filaments of tobacco mosaic virus(TMV) acquired under different conditions. As mentionedin the introduction, the performance of the proposedapproach is measured in terms of the percentages ofcorrectly identified filaments, false dismissals and falsealarms, as compared to the filaments that would have beenselected by an experienced user. Table 1 summarizes theoverall performance of the proposed approach as appliedto the detection of TMV filaments during 3 independentacquisition sessions. Training data was obtained from 141high magnification images collected in a separate session.The training data is used to derive high-level knowledge,

such as range of filament width in pixels, threshold fordetermining parallelism of two line segments, and so on,which is required by our grouping algorithms. The totalnumber of filaments in each session, listed in Table 1, wasvisually counted by one of the authors. Table 1 indicatesthat on average ninety-two percent of TMV filaments canbe automatically detected with a very low false alarm rate(lower than four percent, shown in the last row of theTable). There are still a certain number of false dismissals(lower than eight percent, see Table 1). However, whilewe would like to reduce the false dismissals, the yield iscurrently sufficient to produce a 3D map during a singledata acquisition session. Thus we have achieved theimmediate goal of fully automatic filament detection fromimages acquired in defocus pair using cryo-electronmicroscopy.

Table 1. Performance of the proposed approachmeasured in 3 independent sessions.

Session No. 1 2 3 Avg.Magnification (x) 66K 66K 88KAverage dose(e/A^2)

10.69 11.95 11.87

Defocus value forNTF images (µm)

-0.3 -0.25 -0.15

Imag

eac

quis

itio

nco

nd.

Defocus value forFFF images (µm)

-2 -2 -3

Number of defocus pairsof images

412 444 302

Total number offilaments

1308 592 598

Correctly detectedfilaments (%)

93.83 92.06 91.14 92.34

False dismissals (%) 6.17 7.94 8.86 7.66False alarms (%) 5.51 3.36 1.03 3.30

4. Summary and concluding remarks

The paper presents an accurate approach for automaticidentification of filamentous structures in low-dose, low-contrast images acquired in defocus pair by cryo-electronmicroscopy. A defocus pair consists of a near-to-focus(NTF) image acquired first followed by a far-from-focus(FFF) image. A three-level perceptualorganization algorithm is developed to successfully detectfilaments in the FFF image. At each level, featureelements that are not grouped into higher-level features arediscarded. At the signal level, edges (mostly filamentboundaries) distorted by strong noise are detected usingthe Canny edge detector. At the primitive level, collineardiscontinuous edges are organized into line segments witha complete description using the Hough transformfollowed by detecting end points of line segments. At thestructural level, line segments are grouped into

filamentous structures, the area enclosed by a pair of linesegments, by seeking parallelism and exploiting high-levelknowledge obtained from training data. In the case of twofilaments joining together end to end, a statistical methodis used to accurately locate the splitting boundary and thusseparate them along the boundary. Finally, the NTFimage is aligned to the FFF image using the phasecorrelation technique and filaments in the NTF image aredelineated at the same coordinates identified in the FFFimage.

The performance of the proposed approach has beentested and evaluated by applying it to high magnificationimages of helical specimens preserved in vitreous ice,particularly the tobacco mosaic virus (TMV).Experimental result indicates that on average over ninety-two percent of a large number of filaments can beaccurately detected by the proposed approach with a lowpercentage of false dismissals and a very low percentageof false alarms, which is desirable for the purpose ofstructural reconstruction. The yield is currently sufficientto produce a 3D map during a single data acquisitionsession. In contrast with the conventional cross-correlation the result of the phase correlation alignmentexhibits a much shaper correlation peak. Using the phasecorrelation technique, the alignment between defocuspairs, where there are filaments selected in the FFFimages, can be achieved with almost 100% accuracy.

The proposed systemic practical approach forautomatic detection of filamentous structures should notonly facilitate the three-dimensional reconstruction ofhelical objects for cryo-electron microscopy, but alsoexpedites automated electron microscopy. In our effortstoward high throughput automated electron microscopy,we are next working on exploring machine learningtechniques for automatic selection of good qualityfilaments (for the purpose of reconstruction) from theoutput of the automatic detection stage. We thus need todevelop quantitative ways to measure image quality of thefilaments.

AcknowledgementsThe authors gratefully acknowledge Jim Pulokas and the rest

of the Leginon team at the Beckman Institute for collecting thedata and Ron Milligan at The Scripps Research Institute forproviding the TMV specimen. This project is supported by NSF(DBI-9904547, DBI-9730056) and NIH (GM61939-01).

References[1] Dubochet, J., Adrian, and et al. (1988) Cryoelectronmciroscopy of vitrified specimens, Quart. Rev. Biophys. 21, 129-228.[2] Henderson, R. (1995) The potential and limitations ofneutrons, electrons, and X-rays for atomic resolution microscopyof unstained biological macromolecules, Quart. Rev. Biophys.28, 171-193

[3] Potter, C. S., Chu, H., and et al. (1999) Leginon: A systemfor fully automated acquisition of 1000 micrographs a day,Ultramicroscopy, 77, 153-161.[4] Carragher, B., Kisseberth, and et al. (2000) Leginon: Anautomated system for acquisition of images from vitreous icespecimens, J. Struct. Biol.132, 33-45.[5] DeRosier, D. J., and Moore, P. B. (1970) Reconstruction of

three-dimensional images from electron micrographs ofstructures with helical symmetry, J. Mol. Biol. 52, 355-369.[6] Unwin, P. N. T., and Klug, A. (1974) Electron microscopy ofthe stacked disk aggregate of tobacco mosaic virus protein. I.Three-dimensional image reconstruction, J. Mol. Biol. 87, 641-656.[7] Stewart, M., Kessler, M. W., and Levine, R. J. C. (1981)Structure of Limulus telson muscle thick filaments, J. Mol. Biol.153, 781-790.[8] Wagenknecht, T., DeRosier, D. J., Shapiro, L., andWeissborn, A. (1981) Three-dimensional reconstruction of theflagellar hook from Caulobacter crescentus, J. Mol. Biol. 151,439-465.[9] Egelman, E. H. (1986) An algorithm for straightening imagesof curved filamentous structures. Ultramicroscopy 19, 367-374.[10] Milligan, R. A., and Flick, P. F. (1987) Structuralrelationships of actin, myosin, and tropomyosin revealed bycryo-electron microscopy, J. Cell Biol. 105, 29-39.[11] Bluemke, D. A., Carragher, B., and Josephs, R. (1988) Thereconstruction of helical particles with variable pitch,Ultramicroscopy 26, 255-270.[12] Carragher, B., Bluemke, D. A., et al. (1988) Structuralanalysis of polymers of sickle cell hemoglobin. I. Sicklehemoglobin fibers, J. Mol. Biol. 199, 315-331.[13] Bremer, A., Henn, C., Goldie, and et al. (1994) Towardsatomic interpretation of F-actin filament three-dimensionalreconstruction, J. Mol. Biol. 242, 683-700.[14] Whittaker, M., Carragher, B. O., and Milligan, R. A. (1995)PHOELEX: A package for semi-automated helicalreconstruction, Ultramicroscopy 58, 245-259.[15] Owen, C. H., Morgan, D. G., and DeRosier, D. J. (1996)Image analysis of helical objects: The Brandeis helical package,J. Struct. Biol. 116, 167-174.[16] Nogales, E., and Grigorieff, N. (2001) Molecular machines:Putting the pieces together, J. Cell Biol. 152, F1-F10.[17] Fuller, S. D. (1987) Cell 48, 923.[18] Schrag, J. D., Prasad, B. V. V., Rixon, F. J., and Chiu, W.(1989) Cell 56, 651.[19] Zhou, Z. H., and Chiu, W. (1993) Prospects of using anIVEM with FEG to record images towards atomic resolution,Ultramicroscopy 49, 407-416.[20] Zhou, Z. H. (1995) High resolution 3-Dimensional electroncryomicroscopy and reconstructions of herpes simplex viruscapsids. Ph.D. thesis, Baylor College of Medicine.[21] Jeng, T.-W., Crowther, R. A., Stubbs G., and Chiu, W.(1989) Visualization of Alpha-helices in tobacco mosaic virus bycryo-electron microscopy, J. Mol. Biol. 205, 251-257.[22] Lowe, D. G. (1985) Perceptual Organization and VisualRecognition, Boston: Kluwer Academic Publishers.[23] Koffka, K. (1935) Principles of Gestalt psychology,Harcourt, Brace and World, 1935.[24] Kohler, W. (1947) Gestalt psychology, Liveright, NewYork, 1947.

[25] Mohan, R. and Nevatia, R. (1992) Perceptual organizationfor scene segmentation and description, IEEE Trans. Patt. Analy.And Machine Intell.14, 616-634.[26] Lowe, D. G., (1987) Three–dimensional object recognitionfrom single two – dimensional images, Artificial Intelligence, 31,355-395.[27] Sarkar, S., and Boyer, K. L. (1993) Integration, inference,and management of spatial information using bayesian networks:perceptual organization, IEEE Trans. Patt. Analy. And MachineIntell.15, 256-273.[28] Shi, J., and Malik, J. (2000) Normalized cuts and imagesegmentation, IEEE Trans. Patt. Analy. And Machine Intell.22,888-905.[29] Palmer, S. E. (1983) The psychology of perceptualorganization: A transformational approach, In J. Beck, B. Hope,and A. Rosenfeld, editors, Human and Machine Vision, 269-339,New York: Academic Press.[30] Sarkar S., and Boyer, K. L. (1994) Computing PerceptualOrganization in Computer Vision, Singapore: World Scientific.[31] Williams, L. R., and Thornber, K. K. (1998) A comparisonof measures for detecting natural shapes in clutteredbackgrounds, in Proceedings, 5th European Conf. ComputerVision, Vol. 2, 432-48.[32] Canny, J. (1986) A computation approach for edgedetection, IEEE Trnas. Patt. Analy. and Machine Intell. 8, 679-698.[33] Hough, P.V.S. (1962) Method and means for recognizingcomplex patterns, U.S.Patent 3,069,654.[34] Trucco, E., and Verri, A. (1998) Introductory techniques for3-D computer vision, New Jersey: Prentice Hall.[35] Heath, M. D., Sarkar, S. et al. (1997) A robust visualmethod for assessing the relative performance of edge-detectionalgorithms, IEEE Trnas. Patt. Analy. Mach. Intell.19, 1338-1359.[36] Duda, R. O., and Hart, P. (1972) Use of the Houghtransformation to detect lines and curves in pictures,Communications of the ACM, 15, 11-15.[37] Yamato, J., Ishii I., and Makino, H. (1990) Highly accuratesegment detection using Hough transformation, Sys. Comp.Japan, 21, 68-77.[38] Atiquzzaman, M., and Akhtar, M.W. (1994) Complete linesegment description using the Hough transform, Image andVision computing 12, 267-273.[39] Karmat-Sadekar V., and Ganesan, S. (1998) Completedescription of multiple line segments using the Hough transform,Image and Vision Computing 16, 597-613.[40] McLaughlin, R. A., and Alder, M. D. (1997) Technicalreport – the Hough transform versus the Upwrite, Tech. Rep.TR97-02, Univ. of Western Australia, CIIPS, Dept. of Electricaland Electronic Eng.[41] Frank, J. (1980) The role of correlation techniques incomputer image processing, Computer processing of ElectronMicroscope Images (ed. By P.W. Hawkes), 187-222, Berlin:springer.[42] Al-Ali, L., and Frank, J. (1980) Resolution estimation inelectron microscopy, Optik 56, 31-40.[43] Saxton, W. O. (1994) Accurate alignment of sets of images,J. Microscopy 174, pt 2, 61-68.[44] Kuglin C. D. and Hines D. C. (1975) The phase correlationimage alignment method, Proc. of Int’l Conf. Cybernetics andSociety, 163-165.