Scanning Optical Mosaic Scope for Dynamic Biological Observations and ManipulationBenjamin Potsaid, Yves Bellouard1, John T. Wen

Center for Automation TechnologiesRensselaer Polytechnic Institute, Troy, NY, USA

Abstract. Conventional microscopes with high magnification are limited by the size of the field of view.For biological observations and in particular for biological specimen manipulation, the need for a larger fieldof view has recently gained significant attention. For instance, for histological sections, specimens are ratherlarge and need to be observed in their entirety with an high enough level of magnification. In cell biology,there is also a need for tracking multiple events occuring in a cell population and therefore, it is necessaryto be able to look at different portions of a specimen at the same time. Enlarging the field of view has beendone so far, by moving the specimen under the objectives. However, in general, the moving stage approachhas two drawbacks: low-speed operation and time consuming calibration to gain accuracy. It also requiressophisticated, time consuming, vision processing algorithm to stitch grabbed images together. This paperintroduces an alternate approach that consists in combining high speed scanners and a high speed camerato create a mosaic view over a large field of view at high refresh rates. We state the design problem of theoptical, motion, and image processing subsystems and discuss the initial experimental setup and results. Asa typical application, we emphasize the usefulness of this approach for cell population observations.

1 IntroductionFor biology and more specifically for biological micro-manipulation, a microscope is an indis-pensable tool. Light microscopy has proven to be and is becoming an even more indispenabletool in biology. Indeed, major biological discoveries are often the result of new observationsmade possible by substantial progress and improvements in microscopes [1-6]. However, con-ventional microscopes suffer from the limitation that high magnification reduces the size of thefield of view. As a result, many micro-manipulation tasks that requires micron to sub-micronsprecision over millimeter work volume are beyond the capability of fixed optical microscopes.

For biological observations and in particular for biological specimen manipulation, the need fora larger field of view has recently gained significant attention. For instance, for histological sec-tions, specimen are rather large and needs to be observed in their entirety with an high enoughlevel of magnification. Another important application of microscopes in biology, is the study ofcellular processes (e.g. mitosis, apoptosis), or the interaction between two cells (e.g. parasitic,bacterial or viral attachment) [7-10]. Events of interest typically occur only in a small percentage(<2%) of a population at any one time, therefore, a method of selecting a particular cell from alarger population is required. In particular, it is highly desirable to observe multiple events at thesame time which requires a global observation capability of the specimen.

Until now, the common approach to these problems has been to introduce a moving stage underthe microscope. With the advance of digital image processing and real-time computation, thestage motion can be guided by the observed image to automatically position the specimen [11,

1. Corresponding / presenting author : Dr Yves Bellouard, Center for Automation Technologies,Rensselaer Polytechnic Institute, CII 8211, 110 8th Street, Troy, NY 12180-3590 USA. Tel :(518) 276 6696 / Fax : (518) 276 4897 - email : bellouard@cat.rpi.edu

12]. However, in general, the moving stage approach has two drawbacks: motion of the stage anduneven distribution of the heat between the stage, slide, and light source, always introduce unde-sirable disturbances to the specimen, and the movement of the stage needs to be sufficiently ac-curate for the proper placement of the specimen. To address these issues, the stage usuallyoperates at low speed, and requires costly instrumentation, precision mechanisms, and time con-suming calibration to gain accuracy. With these constraints, tracking of multiple events simulta-neously is either not done or performed at very low rate, preventing collection of populationstatistics such as the spatial distribution and temporal frequency of the event of interest.

To address the limitation of field vision and more specifically, to enable automated image guidedcell analysis, we propose a new concept of microscopy based on an image mosaic. The key ideais to combine high-speed scanning mirrors with a high-speed camera to create an effective en-larged image [13]. The composite image can be processed on–the–fly to automatically detect cellevents of interest and track and record multiple asynchronous like for instance, cell events whilemonitoring the overall cell population.

Our approach offers key advantages compared to the current moving stage technology: scanningmotion and specimen are decoupled resulting in disturbance-free observation, and multiple cellevents can be automatically detected and tracked for increased efficiency and information on cellpopulation statistics. Since the scanning mirrors are placed between the objective lens and cam-era, the position uncertainty is not magnified as is the case for moving stages. This reduces theequipment cost as well as the calibration time.

In this paper, we present this new concept of microscope as well as some preliminary results ob-tained in micro-manipulation and cell population observations.

2 Microscope DesignThe scanning optical mosaic scope (SOMS) consists of a two-dimensional galvonometer (galvo)with a mounted mirror (or two one-dimensional galvos with mounted mirrors) scanning over thespecimen in a prescribed pattern, shown schematically in Figure 1. At each image sampling point,the scanning mirrors come to a brief stop and a complete image is acquired with a high speedcamera. The multiple images are then stitched together to form a single mosaic image which canbe displayed or used to guide the tasks under the microscope.

Many other variations on this design are possible, for example, using a two-axis single mirrorscanner [14,15]. The optical system can be divided into two main blocks:

• Block for scanning: In its simplest form, this block should include the objective lens, thescanning mirrors and the iris. In the example shown in Figure 1, the object is placed atthe focal plane of an achromat lens (L1) such that each ray reflecting off the object iscollimated at the scanning mirrors. An iris is placed just after the lens to enhance theimage contrast.

• Block for image capture: The second block conditions the image according to therequired performance such as the desired magnification and the CCD size. In our exam-ple, a Galilean-like optical system (5 times ratio) is used, which consists of a converginglens (L2) and a diverging lens (L3). Rays exit parallel to the Galilean expander and forman image on the CCD by means of a fourth lens (L4).

Other optical systems such as a laser beam [15] can be added between the two optical blocks toperform different functions like machining or fluorescent stimulation. Figure 1 also shows thekey design parameters of SOMS. On the optical side, the size, shape and distance from the mirrorof the lens L1, will directly affect the size of the field of vision as well as the image quality (suchas optical aberrations, image distortions, etc.) as the system is working off the optical axis. Onthe mechanical side, the settling time of the scanner will define the refreshing rate. Many of theseparameters are closely related. For example, a larger mirror will allow, among others, a larger fieldof vision but at the cost of a longer settling time and therefore a lower refreshing rate.

Forming a larger image through mosaicking is a well established technology, from NASA plan-etary fly-bys to the photo-stitching software in consumer digital cameras. Confocal micro-scopesalso employ high speed scanning to form images but only a single pixel data is obtained at eachscan [16-18]. The use of precision galvanometers (also called galvos), commonly employed forlaser marking applications, allows image mosaiking without registration. To be specific, the gal-vos that we use (from Cambridge Technology, MA) have the angular resolution of 0.0003 (20motion encoded by 16 bits). Using the actual lens selection as described in Figure 1, this translatesto 0.5 micron resolution in the object plane. This could easily be lowered to sub-microns resolu-tion since it only depends on the focal distance of the lens L1. In contrast, to achieve the sameresolution with moving stages, very expensive instrumentation is needed (in terms of encoderresolution as well as guiding technology).

With the target refresh rate of 25Hz, the timing budget for a complete image is 40ms. The maincontributors to the time consumption are scanner motion (including motion profile generationand physical movement from one scan area to the next), and image acquisition and processing.These two operations can be executed in parallel since they involve different processors. Basedon the commercially available technologies for galvanometer and its controller, high speed cam-

Lens L1

Lens L2Lens L3

Lens L4

CCD

Mirrors

θx θy

f=50.0 mmD=25.4 mmAchromat doublet

f=50.0 mmD=25.4 mmAchromat doublet

f=-9.0 mmD=9.0 mmPlano-concave f=100.0 mm

D=15.0 mmBiconvex

Object placed at the focal plane

Iris

6.6 x 8.8 mm

Galilean lens assembly

Iris

Galvanometers

Figure 1. Overview of current experimental setup. The optical system can be divided into two mainblocks - the scanning part (including objective lens (L1), scanning mirrors, and the iris) and the imagecapture block (a Galilean-like optical system (L2-L3), an imaging lens (L4) and a CCD camera.

era and frame grabber, and computer, the bottleneck of the system appears to be the imageprocessing.

Current implementation can achieve the full image update rate of about 1Hz. With currentlyavailable technology, it is feasible to attain 25Hz refresh rate for such tasks at a moderate cost.In terms of optical resolution, there needs to be an order of magnitude increase in order to meetthe requirement of biologists. In order to achieve this, the optical design problem in the pro-posed research needs to be tackled.

As mentioned earlier, current practice of achieving a large field of view relies primarily on movingthe specimen under the microscope using an x-y stage. This approach requires costly high reso-lution sensors and slow motion to ensure positioning accuracy in the image plane. SOMS effec-tively addresses both of these limitations. Since the only moving part is the light weight scanningmirrors, there will be no disturbance to the specimen and high speed simultaneous tracking ofmultiple cell events is easily attainable. As for the positioning accuracy, since the scanning mirrorsare placed after the objective lens in the “infinity space,” (the space where rays are parallel, be-tween L1 and L2 on Figure 1), the accuracy of the mirror position can be as low as the imageresolution. In contrast, the position error of the stage is magnified through the entire optical sys-tem. This gain in image plane accuracy allows the acquired images to be tiled together withouttime consuming image alignment.

3 Experimental setup and preliminary results

3.1 Micromanipulation

We have constructed a low-cost proof-of-concept testbed to demonstrate the optical scanningprinciple and image mosaicking capability for micromanipulation. A millimeter-size Shape Mem-ory Alloy-based gripper [19] was mounted on a commercial cartesian X-Y-Z micromanipulatorand was used to performed a typical assembly-like task under the supervision of a SOMS proto-type.

For this initial setup, the scanner operates at a low speed to allow ample time to settle, the opticalsystem has only minimal complexity, and image acquisition is based on freely available software.For the ease of development, we use MATLAB to coordinate the motion control with the imageacquisition and image processing. As a result, the bulk of the operation time at the present isspent on image acquisition and processing.

The proposed task - shown on Figure 2 - was to move the gripper to a defined location A, toperform a grasping/releasing operation (grasping of a thin aluminum foil) and then to move thegripper to a location B. For this simple proof-of-concept, the task space was chosen to be a planesince the focusing lens of the SOMS’ prototype is not motorized. However, Having the appro-priate equipment would allow to consider 3D task space with the caveat that proper illuminationand space configuration should be chosen to avoid possible shadowing effect.

The dimension of each imaged area is 2mm×1.5mm and the overall field of view is 8mm×6mm.Therefore, the task space was entirely covered by the microscope overall field of view. For thisexample, the resolution was 6.25×6.25 µm2/pixel which is currently limited by the frame grabberused (resolution: 320×240).

For computer assisted task-supervision, a mobile window, defined by a matrix of 3 x 3 contigu-ous tiles is acquired (Figure 3). Some basic image processing and Kalman-filter based trackinghas been implemented in order to keep the micro-gripper tip in the center of the mosaic imageas the micromanipulator moves. Two additional fixed-tiles defined as the task objectives are alsocontinuously updated during the manipulation.

SOM

S ac

cess

ible

fie

ld o

f vi

sion

(foca

l pla

ne /

task

pla

ne)

X

YZ

Micromanipulator(cartesian)Mobile window

(3 x 3 tiles)

B

A

Fixed tiles(Task objectives)

Task supervision(computer)

A B

Figure 2. A vision based assembly task using the SOMS. In this example, the task was to bring thegripper to point A, do a grasp and release, and then moved to point B. A mobile window defined by amatrix of 3 x 3 contiguous tiles is acquired and automatically adjusted to keep the gripper in the middlewhen the micromanipulator is moving. Two fixed tiles defined as the task objectives are also continu-ously updated during the manipulation.

Figure 3. Video footage of an assembly task performed under SOMS. The 3x3 matrix of tilesare implemented in order to keep the micro-gripper tip in the center. The two additional tiles arefixed. All the tiles are continuously refreshed. The gripper first starts at a position next to a wire(1) and moves to the right and grasps an aluminum foil (6).

Figure 3 shows a video footage of an micromanipulation performed under SOMS. The sequenceis as follow : 1. The gripper starts above the 100 micron diameter wire. Note that the gripper tip is visible in

both the fixed tile on the left as well as the 3x3 mosaic. 2. As the gripper moves upwards and towards the right, the image processing routine keeps

the gripper centered in the 3x3 mosaic. 3. The wire and aluminum foil can be seen in the workspace4. The gripper is now positioned directly above the aluminum foil. 5. The gripper approaches the aluminum foil and enters in the field of view of the fixed tile on

the right. 6. The gripper grabs the aluminum foil.

Because of the use of a low-cost frame-grabber and of a conventional camera, the refreshing ratewas limited to one Hertz. However, this simple proof-of-concept demonstrates the possible useof SOMS for micro-assembly. Critical aspects that needs to be addressed carefully are the illumi-nation through the entire space and also possible shadowing effect depending on the task spaceconfiguration. As mentioned earlier, a z-motorized objective would allow to address 3D taskspace.

3.2 Multiple cells events monitoring

To demonstrate the potential of SOMS in biology, we have focused our effort on the observationof cell population. For these specific observations, an inverted microscope configuration is pre-ferred (see Figure 4). The overall principle and optical configuration remains similar to the onepreviously shown on Figure 1. In this configuration, the light comes from the top through anillumination lens. A white source (Halogen) was used with various color filters to optimized theimage quality.

The study of cellular processes (e.g. mitosis, apoptosis), or the interaction between two cells (e.g.parasitic, bacterial or viral attachment) [7-10] is an important application of microscopes in biol-ogy. Events of interest (Figure 5 illustrates an exemple of mitosis) typically occur only in a small

Figure 4. Prototype of SOMS setup as an inverted microscope for biological applications.

percentage (<2%) of a population at any one time, therefore, a method of selecting a particularcell from a larger population is required. While there are methods for increasing the frequencyof some of these events, such as cell synchronization [20,21], there are many associated problemswith doing so: many cells types do not fully respond to synchronization, there is only a small timewindow to image the cellular event and a non-physiological environment is created under the mi-croscope. Currently, a user of a light microscope will scan a population “hunting” for cells thatare either in the process of interest or transitioning into the process of interest.

To perform this scanning operation, the specimen is placed on an x-y moving stage, which is thenmanually or automatically moved until the portion of interest is underneath the objective. As stat-ed earlier, optical microscopes have an intrinsic limitation: high magnification reduces the size ofthe field of view. It is therefore necessary to first observe a large cell population to detect theinception and location of a cell event of interest using a low-magnification objective (to gain alarge field of view), then move the specimen (using the stage) until the desired cell is positionedunder the high-magnification objective. Due to the imprecision of the stage, there is typicallymuch switching back and forth between high-mag and low-mag objective lenses and subsequentrefocusing of the microscope before the specimen can be accurately positioned. This is not onlylaborious and time-consuming, but can require substantial training to be able to detect the earlieststage chromosome condensation (prophase), or the cellular attachment on some parasites [10],bacteria [8] or HIV-1 particles for instance [9]. Additional problems associated with the methodare that only one cell is followed at a time, and the transition from one stage to another, transitionfrom G2 to mitosis, for example, can be very dicult to detect even for very experienced micro-scopists.

While sequential filming of cells can somewhat overcome the first problem (one cell at a time),the second problem (transition between stages) cannot be so easily addressed.

For this particular application, SOMS effectively addresses these limitations. Since the only mov-ing part is the light weight scanning mirrors, there is no disturbance to the specimen and highspeed simultaneous tracking of multiple cell events is easily attainable.

Figure 6 shows a population of Eukaruotic cells observed with the SOMS prototype shown onFigure 4. The picture shows a 3x3 mosaic of tiles and two individual tiles illustrating the capabilityto scan over non-adjacent user-defined as “region-of-interest”. This illustrates non only the ca-pability of the SOMS to continuously refresh a enlarge field of view made of a matrix of connect-ed tiles but also the capability to scan over non-adjacent tiles user-defined as “region-of-interest”.The resolution of individual tiles is 580 x 360 using a Matrox frame grabber and corresponds toan area of about 400 x 300 microns on the specimen. The magnification ratio is about 15 times.

In a preliminary experiment, dynamic observations of living cells were performed over a periodof several hours. Grabbed images were stored every minute in order to limit the allocated mem-ory.

Figure 5. Example of Cell Event of Interest : Mitosis. When mitosis occurs in a cell, there willbe gradual changes of shape, area, centroid, contrast, and density of the cell.

As an example, a portion of a video footage is shown on Figure 7. A possible mitosis event canbe seen on the lower left portion of the middle of the picture. Unfortunately, the contrast andthe magnification level are not enough high to be fully conclusive.

4 Future workAlthough we have demonstrated the principle of Scanning Optical Mosaic Scope for micro-ma-nipulation and biological observations, this technology is still at its infancy. Additional work haveyet to be done. In particular for some applications, the actual magnification will have to be in-

Figure 6. Eukaruotic cells grown on a microscope slide and observed using the current “low-resolu-tion” SOMS prototype. The upper part of the pictures shows a 3 x 3 matrix of tiles. Each individual tilehas a resolution of 580 x 360 pixels which translates, with the current lens selection shown on Figure 1,into a region of approximatively 400 x 300 microns on the specimen. The bottom part of the picturesshows two non-adjacent tiles grabbed from two user-defined regions (A and B). This illustrates not onlythe capability of the SOMS to continuously refresh a enlarge field of view made of a matrix of connectedtiles but also the capability to scan over non-adjacent tiles user-defined as “region-of-interest”.

Figure 7. Portion of a video footage of Eukaruotic living cells observed with SOMS. Theelapsed time between each frame is about 25 min. A possible mitosis event is emphasized(arrow). However, the contrast and the magnification level are not enough high to be fullyconclusive.

creased. The image quality also needs some improvement. The overall design consideration ofSOMS requires a multidisciplinary approach and optimization that includes optical subsystemperformance (e.g., magnification, depth of field, spherical and coma aberrations, astigmatism,and distortion), mechanical subsystem performance (e.g., velocity and acceleration limits, settlingtime, and positional accuracy of the scanning mirrors), and image acquisition subsystem per-formance (e.g., image acquisition rate, pixel addressability, feature recognition and tracking capa-bility).

As an illustration, the motion of the mirrors changes the projected aperture and therefore alsothe amount of collected light. The optical design therefore needs to take these multitude of de-sign considerations into account. The operation of SOMS requires extensive imaging at the outerportion of the objective lenses (off-axis imaging) which increases the influence of higher ordereffects such as distortion effects and aberrations. A key objective of the optical design is toachieve high image quality and uniformity over the full range of the scanner motion.

AcknowledgementsThe authors would like to thank Dr Jacques Izard and Dr Richard Cole from Wadsworth Re-search Center (New-York State Department of Health) for their useful comments and insightregarding cells events detection and for providing us with specimens of living cells. We alsowould like to thanks NYSTAR for the financial support of part of this preliminary work.This research is supported in part by the National Science Foundation under Grant 0301827.

References

1. B. Hermann and J.J. Lemasters. Optical Microscopy: Emerging Methods and Applications.Academic Press, New York, NY, 1993.

2. E.M. Slayter and H. S. Slayter. Light and Electron Microscopy. Cambridge University Press,Cambridge, UK, 1998.

3. M. Davidson and M. Abramowitz. Optical microscopy. Olympus America, Inc., Melville,New York, 1998.

4. F. Zernike. Physica, 9:686–693, 1942.5. G. Nomarski. Nouveau dispositif pour l’observation en contraste de phase différentiel. J.

Phys. Radium, 16:9S–11S, 1955.6. R. Hoffman. J. Microscopy, 110:205–222, 1977.7. R.W.E. Clarkson, J.L. Heeley, R. Chapman, F. Aillet, R.T. Hay, A. Wyllie, and C.J.Watson.

NF-B inhibits apoptosis in murine mammary epithelia. J. Biol. Chem, 275, April 2000.8. T.J. Fitzgerald. Attachment of treponemes to cell surfaces. In R.F. Schell and D.M. Musher,

editors, Pathogenesis and immunology of treponemal infection, pages 195–228. MarcelDekker, Inc., New York, NY, 1983.

9. P.R. Clapham, A. McKinght, HIV-1 receptors and cell tropism, Br. Med. Bull., 58:43-59,2001.

10. D. Soldati, J.F. Dubremetzb, and M. Lebrunc. Microneme proteins: structural and functionalrequirements to promote adhesion and invasion by the apicomplexan parasite toxoplasmagondii. International Journal for Parasitology, 31:1293–1302, 2001.

11. F. Ianzini, L. Bresnahan, L. Wang, K. Anderson, and M.A. Mackey. The large scale digital cellanalysis system and its use in the quantitative analysis of cell populations. In 2nd Annual Int.IEEE-EMBS Special Topic Conf. on Microtechnologies in Medicine & Biology, pages 469–475, Madison, WI, 2002.

12. P.C. Goodwin and A. Kwiatkowski. Absolute positioning control. applications in imaging,pharmacological delivery, and functional genomics with live cells. G.I.T. Imaging & Micros-copy, 2:50–52, 2000.

13. B. Potsaid, Y. Bellouard, J.T. Wen, Scanning Optical Mosaic Scope for Micro-Manipulation,3rd International Workshop on Microfactories, Minneapolis, USA, Sept. 16-18, Ed. Nelson/Hollis, pp.85-88, 2000.

14. M. Hafez. Compact Fast-steering tip/tilt laser scanner for high power material processing ap-plications. PhD thesis, EPFL, Lausanne, Switzerland, 2000.

15. Y. Bellouard, T. Lehnert, R. Clavel, T. Sidler, and R. Gotthardt. Laser annealing of shapememory alloys: a versatile tool for developing smart micro-devices. J. Phys. IV France,11:Pr8-576, 2001.

16. S. Inoue. Foundations of confocal scanned imaging in light microscopy. In James Pawley, ed-itor, The handbook of biological confocal microscopy, pages 1–13. IMR Press, Madison, WI,1989.

17. O. Trepte and A. Liljeborg. Computer control for a galvanometer scanner in a confocal scan-ning laser microscope. Optical Engineering, 33(11):3774–3780, 1994.

18. G. C. Loney. Scanner component and head development for confocal microscopy usingmoving mirror technology. In Proc. of SPIE Vol. 1987, pages 129–136, 1993.

19. H. Zhang, Y. Bellouard, T. Sidler, E. Burdet, A.-N. Poo, R. Clavel, "A monolithic ShapeMemory Alloy Microgripper for 3-D Assembly of Tissue Engineering Scaffolds", Microro-botics and Microassembly III, Bradley J. Nelson, Jean-Marc Breguet, Editors, Proceeding ofSPIE, Vol. 4568, pp.50-60 (2001).

20. G.F. Merrill. Cell synchronization. Methods in Cell Biology, 57:229–249, 1998.21. L.J. Marnett and J.A. Pietenpol. Cell cycle re-entry following chemically-induced cell cycle

synchronization leads to elevated p53 and p21 protein levels. Oncogene, 15(22):2749–2753,1997.