Proc. Natl Acad. Sci. USAVol. 78, No. 11, pp. 6927-6931, November 1981Cell Biology

Computer-enhanced video microscopy: Digitally processedmicroscope images can be produced in real time

(image processing/phase-contrast optics/differential interference-contrast optics/tissue culture)

ROBERT J. WALTER AND MICHAEL W. BERNSDepartment of Developmental and Cell Biology, University of California, Irvine, California 92717

Communicated by Keith R. Porter, July 21, 1981

ABSTRACT Digital processing techniques can be used togreatly enhance the available information in an optical image.Although this technology has been routinely used in many fieldsfor a number ofyears, little application ofdigital image-processingtechniques have been made toward analysis and enhancement ofthe types of images seen most often by the research biologist. Wedescribe here a computer-based video microscope system that iscapable of performing extensive manipulation and enhancementof microscope images in real time. The types of manipulationspossible with these techniques greatly surpass the enhancementcapabilities of photographic or video techniques alone. The speedand flexibility of this system enables experimental manipulationof the microscopic specimen based on its live processed image.These features greatly extend the power and versatility ofthe lightmicroscope.

The need to record and analyze poor-quality, low-contrast mi-croscope images is a frequent problem confronting the cell bi-ologist. Although contemporary investigators have available tothem a wide variety of optical techniques (phase contrast, po-larization, dark field, interference contrast, etc.) even the mostcareful use of these techniques may still fail to produce a sat-isfactory image from a given specimen. These image limitationsare particularly severe when the microscope is used as a toolfor actual experimentation. Microirradiation or micromanipu-lation of material viewed through the microscope can be so re-stricted by image quality that an otherwise feasible experimentcan be rendered impossible.

Optical images that are unsatisfactory in their raw form maystill contain useful information that is unseen due to perceptuallimitations of the human eye. This hidden information can berevealed through the use of various image enhancement tech-niques that range in complexity from simple photographic pro-cessing to sophisticated digital processing routines. In recentyears video technology also has been applied to light microscopyas a method for routinely enhancing microscope images (1, 2).Video cameras with different light-transfer properties can beselected for a given application in the same manner as one wouldselect a photographic film, with the additional advantage thata video camera will also produce an enhanced image in realtime. Sophisticated electronic processing can also be performedon a video signal to reduce noise, subtract background, or makequantitative measurements. Highly light-sensitive video cam-eras can be used to produce bright images from very weaklyilluminated microscope specimens, enabling continuous obser-vation of living material that is adversely affected by the usualhigh level of illumination (3). Although this technology is stillnew, recent advances in video microscopy, such as the detectionof transient low-level subcellular fluorescence (2, 4), and the

development of techniques for enhancing differential interfer-ence-contrast (DIC) and polarizing microscope images (5-7)may be the beginnings ofa real revolution in optical microscopy.The most sophisticated enhancement and analysis of micro-

scope images is obtained through the use of digital processingtechniques on a previously digitized image. This digital imagecan originate from either a converted video signal, a photodiodescanner, or an optical multichannel analyzer. Processing rou-tines such as image subtraction, edge enhancement, spatial fil-tering, and contrast enhancement can be performed on a dig-itized image in ways that far surpass the types of manipulationspossible with photographic or video techniques alone.

Although digital image processing techniques have been ap-plied to biomedical problems for more than 20 years (8, 9), theprimary emphasis of this work has been toward automation ofroutine clinical work such as the classification of cell types,screening of histological samples, or preparation of karyotypes(10-14). Relatively little application of digital processing tech-nology has been made toward enhancement of live microscopeimages of the type most often seen by the research biologist.Although several different microscope image analyzers are com-mercially available, these machines have been designed pri-marily for maximum speed when performing routines with di-rect clinical applications, with a subsequent loss of speed andversatility when they are programmed for other tasks (15). Thisfact, in addition to the cost and complexity of the equipment,has generally inhibited the application of digital image-pro-cessing techniques to real-time microscopy in basic biologicalresearch.We report here the development of a versatile computer-

based video microscope system that is capable of performingextensive and sophisticated image-enhancement routines onmicroscope images in real time. The images produced by thissystem appear to have better contrast and resolution than thoseproduced by conventional microscopy. This system is flexibleand interactive with the microscopist. Examples are presentedof contrast enhancement, edge enhancement, and backgroundsubtraction of microscope images produced from a live videosignal by using digital techniques.

MATERIALS AND METHODS

A schematic diagram of the image-processing system is shownin Fig. 1.

Optics. Optical images are produced by a Zeiss AXIOMATmicroscope in the inverted mode. Specimens can be viewedunder phase contrast, polarization, differential interferencecontrast, and other optical systems. The microscope controlsinclude a 0.5-Am-step, XY-motorized, stage translational con-

Abbreviations: ITF, intensity transformation function; DIC, differentialinterference contrast.

The publication costs ofthis article were defrayed in part by page chargepayment. This article must therefore be hereby marked "advertise-ment" in accordance with 18 U. S. C. §1734 solely to indicate this fact.

6927

6928 Cell Biology: Walter and Berns

LSI/1 I DUALMICROCOMPUTER FLOPPY DISKS

FIG. 1. Schematic diagram of the image-processing system. Im-ages from the AXIOMAT microscope are converted to video format,then routed simultaneously to a black and white monitor and the dig-itizer of the image processor. Operation of the image processor is con-

trolled by the LSI-11 microcomputer using data and programs storedon floppy disks. Processed images are reconverted to video format andviewed on a color monitor. Live video images can be stored on the videotape recorder (VTR) either before or after digital processing. DAC, dig-ital to analogue converter.

trol and an automatic autofocusing device that operates off ofthe video signal. Both devices can be controlled by the com-

puter and image processor. Microscope images can be projectedonto the video camera and viewed through the ocularssimultaneously.

Video Camera. The video camera (model LST-1, Sierra Sci-entific Products, Mountain View, CA) consists of a 1-inch (2.5-cm) chalcogenide (Newvicon) video tube plus rack-mountedcontrol electronics. The sensitivity of this tube is somewhatbetween that of a silicon intensified-target camera and a stan-dard vidicon. This camera system has been selected for minimalgeometric and shading distortion in the video signal. The scan-

ning interval at the primary image is several times smaller thanthe resolution of the microscope at all magnifications. The out-put video signal can be optimized for each image with externalcontrols for gain, direct current (dc) offset, and contrast. Thevideo signal is generated in a 525-line format at a rate of 30frames per sec. Output from the camera can be directed si-multaneously to a 13-inch (33-cm) black and white monitor(where it is viewed unaltered), to the input of the image pro-cessor, and to a video tape recorder.

Digitizer. The video digitizer converts each video line into512 sequential eight-bit intensity values. Only the six most sig-nificant bits of each discrete picture element (termed a "pixel")are actually digitized per video frame; however, a true eight-bitimage can be produced in four video frame times by using cu-mulative averaging techniques. Only the first 512 video linesof each video frame are digitized; consequently, each originalvideo frame is converted into a 512 X 512 array of pixels(262,144 pixels total).

Image Processor. The image processor (model IP5500,DeAnza Systems, San Jose, CA) was originally designed foranalysis of LANDSAT satellite images and has been speciallyadapted for use with the optical microscope. The image pro-

cessor contains three separate 512 X 512 X 8-bit memory chan-nels that are configured to correspond to the red, green, andblue primary images of a normal color picture. Special hardwareallows this memory to store and display three different mono-

chrome (black and white) images when a true color image isnot desired. The processor itself is built around two eight-bitarithmetic logic units that can each perform one of 96 differentarithmetic or logical operations on any combination of two in-puts from the three-memory channels or the live digitized video

signal. Each ALU can produce a full 262,144-pixel image in onevideo frame time. Additional hardware features provide thecapability for defining regions of interest in the image, substi-tuting image intensity values using predefined look-up tables,spatially offsetting images in memory, and manipulating thedisplay of individual memory channels. Processed images arereconverted to video format and routed simultaneously to a 19-inch (48-cm) color monitor and the video tape recorder. Anycombination of these manipulations can be performed on thelive video signal in one frame time.

LSI-11/23 Processor Controller. The image processor is adedicated machine language processor that continuously ma-nipulates and displays the data within memory in response tothe preprogrammed settings ofapproximately 250 16-bit controlregisters. The LSI-11/23 microprocessor (Digital Equipment,Maynard, MA) controls the operation of the image processorby performing read/write operations to these control registersand controlling processor timing. FORTRAN and MACRO lan-guage programs are stored on floppy disks or an RL01 hard copydisk. Digitized images can be stored on the floppy disks at therate of two per disk. Complex image manipulations that cannotbe performed efficiently in the image processor alone can beaccomplished in the memory ofthe LSI-11 by transferring small(8000 pixel) sections of the image to the LSI-11. However, thesemanipulations are slower than the image processor alone by 3or 4 orders of magnitude.

RESULTSContrast Manipulation and Enhancement. A photograph of

a PTK2 tissue culture cell taken with phase-contrast optics isshown in Fig. 2A. This image was photographed directly fromthe video monitor and has not been digitally enhanced. A com-puter generated histogram of the distribution of grey valueswithin this image is shown in Fig. 2B. This histogram shows thatall of the grey tones in this image are tightly clustered arounda sharp central peak corresponding to the phase-neutral back-ground, and that many possible grey tones are not representedin the image. This tight distribution is typical of images pro-duced with phase-contrast or DIC optics. The shoulders of thishistogram are produced by the phase-light and phase-dark re-gions of the cell and contain all of the useful information in theimage. The narrow range of grey values in this image and thelarge amount of background result in a low contrast that makesit difficult to perceive fine detail. This condition is a result ofthe fact that any pixel in the image has a high probability ofhaving neighboring pixels with grey values equal to or nearlyequal to its own; consequently, there is low contrast betweenneighboring pixels, and much detail is hidden.

This masking of detail can be overcome by arbitrarily reas-signing the grey values of the phase-light and phase-dark re-gions of the image to different grey values with higher contrast.Fig. 2D shows the image of Fig. 2A after the grey values werereassigned in this manner. These grey value reassignments arespecified by an intensity transformation function (ITF), derivedby the image processor, that maximizes contrast in the desiredregion of the image. This function is analogous to the charac-teristic curve (sometimes called the Hurter-Driffield curve)that is used to describe the light-transfer properties of photo-graphic film or video cameras. One of the advantages of digitalenhancement techniques is that, unlike the light-transfer prop-erties of films and video tubes, this ITF does not have to belinear or even continuous. This extra capability can be used tomaximally enhance contrast in the information-containing re-gions of an image while leaving unaltered or even reducing thecontrast in background regions. The nonlinear ITF that maxi-

Proc. Nad Acad. Sci. USA 78 (1981)

Cell Biology: Walter and Berns

a! , 8 i Li

'U:-V'' 4

FcIG. 2. Microscope images enhanced by using video and digital techniques. (A) Phase-contrast image of a PTK2 culture cell before digital en-hancement. (D) The same image after digital processing to enhance contrast in phase-light and -dark regions. (G) Same image asD, except the greyvalues of the phase-neutral regions have been arbitrarily assigned to zero (black). (B) Histogram of C showing the relative distribution of greyvalues. (E and H) Image histogram ofD and G, respectively, after enhancement. (C, F, and I) Intensity transformation functions of A, D, and G,respectively. Note nonlinear nature of ITFs of I and H. (J) Differential image processed from the phase-contrast image of A. (K) DIC image of thesame field of cells as in A. Arrows in J andK show the direction of optical shear. (Bar in A = 10 sm.)

Proc. NatL Acad. Sci. USA 78 (1981) 6929

A V

6930 Cell Biology: Walter and Berns

mally enhances the contrast of the phase-light and phase-darkregions of Fig. 2A, while simultaneously not affecting the con-

trast of the phase-neutral background is shown in Fig. 2F. Thegrey value histogram of the enhanced image is shown in Fig.2E. Note that more detail of Fig. 2A is revealed after this pro-

cessing. A linear ITF that does not alter image contrast and isequivalent to a photographic film with a degree of contrast (y)of one is shown in Fig. 2C for comparison.

Once the ITF has been derived, the grey value reassignmentsdefined by this function can be loaded into the appropriate reg-isters of the image processor and used to enhance the contrastofeach subsequent video image in real time as it is being writtento processor memory.

Pseudocolor Contrast Enhancement. When all possible greyvalues are contained in an image, contrast cannot be enhancedwithin one range of grey values without reducing contrast over

another range. This drawback can be overcome by using thetechnique of pseudocolor enhancement.

In our system each color image is displayed as the combi-nation of red, green, and blue primary images, each of whichcan vary in intensity over a 256-step range. More than 16 milliondifferent color combinations are possible. The 256 shades ofgrey in a normal monochrome image represent a small subsetofthis larger class ofcolor combinations; consequently, an imagethat utilizes all ofthe possible grey values in a monochrome im-age has many unused color values when considered in the colordomain. Individual grey values in a monochrome image can bearbitrarily converted to new color intensity values without af-fecting the contrast of remaining grey tones. This techniqueonly differs from other contrast-enhancing routines in that in-dividual grey values are assigned to new colors and not simplyto new shades of grey.Edge Enhancement. Preferential enhancement of the edges

and boundaries within an image is a commonly used processingtechnique for detecting objects and suppressing background(16). Fig. 2G shows an edge-enhanced image of Fig. 2A and D.In this image the contrast of the phase-light and -dark regionswas enhanced as in Fig. 2D; however, the grey values ofphase-neutral background regions were arbitrarily assigned to zero

(black). As with the previous examples, the nonlinear ITF thatproduces this image was loaded into the memory of the imageprocessor, and all subsequent video frames were similarly pro-cessed in real time.A more sophisticated technique for producing edge-en-

hanced images is shown in Fig. 2J. This image was produceddirectly from the phase-contrast image of Fig. 2A in a two-stepprocess. First, the digitized image is written into two memorychannels of the image processor so that there are identical im-ages in each channel. Second, the image in one channel is spa-tially offset a small distance in the horizontal direction (corre-sponding to approximately 0.5 ,um in the image), and this offsetimage is subtracted from the first image. A constant value of 128is also added to each pixel to avoid the production of negativevalues. The resulting image is then displayed on the monitoras a monochrome image. This process can be repeated in pipe-line fashion for each subsequent video frame as it is presentedto the processor, consequently the edge-enhanced image is pro-duced in real time.When small spatial offset values are used, the image pro-

duced by this manipulation approximates the partial derivativeofthe image intensity function in the direction ofthe offset (17).Areas within the original image with high spatial frequencies(such as edges and boundaries) will produce large nonzero de-rivatives, whereas slowly varying background shading will besuppressed.DIC microscopy is an optical technique that also detects ob-

ject boundaries by subtracting offset images. The two offset im-ages are created by perpendicularly polarized beams of lightthat are retarded in phase in relation to one another. The twoimages are subsequently "subtracted" when they are recom-bined, and the retarded wavefronts optically interfere. A pho-tograph ofthe same field ofcells taken with DIC optics is shownin Fig. 2K for comparison.

Although the images in Fig. 2J and K appear similar in manyrespects, there are fundamental differences in- the way the im-ages are produced that can make the digital processing tech-nique more desirable than DIC microscopy in many circum-stances. Both techniques produce images that are approximationsof the phase gradient in the original specimen (18). However,specimens that are birefringent or optically rotate polarizedlight may seriously degrade DIC images but will not interferewith phase-contrast images (19). An edge-enhanced image pro-duced from a phase-contrast image under these circumstanceswill not be subject to this degradation. We have also found thatthe artifactual "halo" produced by phase-contrast optics mayactually produce better-defined edges in the differential phase-contrast image than in the DIC image. The halo tends to ex-aggerate the phase gradient at some of the boundaries withinan image by producing a bright ring on one side of a boundaryand a dark ring on the other, with a sharp inflection in intensityat the actual boundary (18). Because the "difference" techniquewe use produces a partial derivative ofthe image intensity func-tion, this exaggerated gradient produces a sharp edge at theactual boundary in the difference image while somewhat re-ducing the actual halo itself.An intense mercury arc is required to produce a DIC image

on the AXIOMAT microscope in order to compensate for lightlosses at the polarizers and Wollaston prisms. This intensesource is often damaging to living material. In comparison, adigitally processed differential phase image is produced withphase-contrast optics and low-level tungsten illumination thatdoes not have this effect. Consequently, the digital techniquemay be preferred for viewing living specimens, even whenequivalent images are produced by both techniques, in orderto avoid damaging the specimen.

DISCUSSIONThe nonlinear nature of digital processing makes it possible tocreate images that generally cannot be obtained with photo-graphic or video techniques alone. Not only can certain partsof an image be preferentially enhanced in relation to others, aswas shown in Fig. 2 A, D, and G, but this nonlinear feature canalso be used to compensate for other sources of nonlinearity inthe system (20). An ITF can be used to correct for a nonlineardigitizer or video camera response in such a way that intensityin the processed image is proportional to the optical density inthe original image. Nonlinearity in the optical images can alsobe corrected by this technique. For example, it is well knownthat object intensity in a DIC or polarization image is a functionofthe sin' ofthe phase-retardation angle ofthe object in relationto the background (18). An ITF can be used to substitute imageintensity values in such a way that intensity is instead a linearfunction of retardation angle. After this processing, differencesin phase retardation between objects can be directly measuredfrom the differences in intensity between them.We have presented here several examples of digitally pro-

cessed microscope images that reveal more information thanwas discernible in the unprocessed video image. The highlyinteractive nature ofour system, and the ability to do processingin real time allows us to quickly select the optimum enhance-ment parameters for each image while the specimen is being

Proc. Nad Acad. SCti. USA 78 (1981)

Proc. NatL Acad. Sci. USA 78 (1981) 6931

viewed. This speed and flexibility in turn makes it possible todo precise manipulation and quantification of material viewedthrough the microscope based on the processed image. Thesefeatures greatly extend the power and versatility of the lightmicroscope.

This research has been supported by National Institutes of HealthGrants HL 15740, GM 23445, and RRO 1192 and by U.S. Air ForceOffice of Scientific Research Grant 80-0062.

1. Willingham, M. C. & Pastan, I. (1978) Cell 13, 501-507.2. Rose, B. & Lowenstein, W. R. (1975) Science 109, 1204-1206.3. Berns, M. W. (1974) Biological Microirradiation (Prentice-Hall,

Englewood Cliffs, NJ), p. 8.4. Schlessinger, J., Shecter, Y., Willingham, M. C. & Pastan, I.

(1978) Proc. Natl Acad. Sci. USA 75, 2756-2663.5. Inoue, S. (1981)J. Cell Biol 89, 346.6. Allen, R. D., Travis, J. L., Allen, N. S. & Yilmaz, H. (1981) Cell

Motility 1, 275-289.7. Allen, R. D., Allen, N. S. & Travis, J. L. (1981) Cell Motility 1,

291-302.

8. Prewitt, J. M. S. & Mendelson, M. I. (1965) Ann. N.Y. Acad. Sci.128, 1035-1053.

9. Lipkin, L. E., Watt, W. C. & Kirsch, R. A. (1965) Ann. N.Y.Acad. Sci. 128, 984-1012.

10. Olson, A. C., Larson, N. M. & Heckman, C. A. (1980) Proc. NatlAcad. Sci. USA 77, 1516-1520.

11. Bowie, J. E. & Young, I. T. (1977) Acta Cytologica 21, 739-746.12. Bradbury, S. (1979)J. Microsc. (Oxford) 115, 137-150.13. Dunn, R. F., O'Leary, D. P. & Kumley, W. E. (1975)J. Microsc.

(Oxford) 105, 205-213.14. Castleman, K. R., Melnyk, J., Frieden, H. J., Persinger, G. W.

& Wall, R. J. (1976)J. Reprod. Med. 17, 53-57.15. Preston, K. (1976) in Digital Processing of Biomedical Images,

eds. Preston, K., Jr. & Onoe, M. (Plenum, New York), pp.47-58.

16. Pratt, W. K. (1978) Digital Image Processing (Wiley, New York),p. 480.

17. Thomas, G. B., Jr. (1968) Calculus and Analytical Geometry(Addison-Wesley, Menlo Park, CA), 4th Ed., p. 498.

18. Allen, R. D., David, G. B. & Nomarski, G. (1969) Z. Wins. Mikr.69, 193-236.

19. Lang, W. (1969) Zeiss Information No. 73, Reprint S41-210.2-5e(Oberkochen, Federal Republic of Germany).

20. Castleman, K. R. (1979) in Digital Image Processing (Prentice-Hall,Englewood Cliffs, NJ), pp. 84-95.

Cell Biology: Walter and- Berns