Immersive Virtual Colonoscopy - Stony Brook University · A. Virtual Colonoscopy Virtual...

4
Immersive Virtual Colonoscopy Kaloian Petkov, Charilaos Papadopoulos, Student Member, IEEE, and Arie Kaufman, Fellow, IEEE Department of Computer Science and CEWIT Stony Brook University Stony Brook, NY 11794-4400, USA [kpetkov,cpapadopoulo,ari]@cs.stonybrook.edu Abstract— Virtual Colonoscopy (VC) is an established non- invasive alternative to optical colonoscopy for colorectal cancer screening. Currently, radiologists examine VC data sets on a workstation computer using a single screen and simple navigation tools. The limited field of view increases examination time and introduces tedious interaction modalities. We propose conducting VC examinations inside an immersive visualization environment, such as the fully enclosed Immersive Cabin (IC) or semi-immersive workstation display configurations. The additional projection surfaces, life-size visuals and stereoscopic rendering in the IC provide the radiologist with a more intuitive navigation experience and allow for better utilization of the direct and peripheral vision to pick up visual cues related to the presence of polyps. As a result, the doctor is able to examine more of the colon surface at any given time, reducing total time for the screening and increasing the sensitivity of the screening procedure. Keywords- Visualization, Virtual Reality, Scientific Visualization, Medical Imaging I. INTRODUCTION Colorectal cancer is a leading form of cancer in the US and results in more than half a million deaths worldwide every year. Existing screening techniques are very invasive, require a lengthy cleansing and as a result, compliance among patients above the age of 50 is low. Virtual colonoscopy [1,2] is a recently developed technique for non-invasive colon cancer screening which is much more comfortable for the patient and eliminates many of the health risks associated with the traditional approach. However, current implementations of Virtual Colonoscopy do not take advantage of recent developments in 3D and immersive display technologies. Immersive Virtual Colonoscopy improves the quality of the examination by allowing for improved shape perception due to the spatial cues produced by stereoscopic images. Also, the doctor is better able to examine surfaces that might otherwise be occluded simply by moving slightly inside the physical constraints of the immersive environment, or by using a very natural gesture-based interface. We present an implementation of Virtual Colonoscopy in the Immersive Cabin (IC) [3], which is a fully-enclosed visualization environment designed to immerse the user in the visual data. The resulting system provides a more natural interface for interacting with the data and greatly enhances the field of view that the user experiences. In addition, stereoscopic rendering allows for better shape detection based on visual depth cues, which is critical for finding polyps in the colon data. We also present a desktop system that offers many of the same advantages for medical examinations while greatly decreasing the significant financial burden associated with deploying the IC. II. RELATED WORK A. Virtual Colonoscopy Virtual Colonoscopy (VC) has been established as an alternative to optical colonoscopy (OC) for colon cancer screening and shown to compare favorably [4,5]. It is non- invasive and does not require the traditional aggressive bowel cleansing of OC. The day before the procedure, the patient follows a low-residue diet and ingests a barium contrast liquid. A VC session involves the acquisition of a computed tomography (CT) scan of the patient’s abdomen. Due to the contrast material, it is possible to perform electronic cleansing on the CT data to remove any leftover stool from the CT slices. However, radiologists may instead elect to follow the traditional protocol for the physical colon cleansing. The colon surface is then extracted from the cleansed CT data using segmentation and it is visualized via volume rendering or other rendering techniques. The radiologist examines the virtual data by following the pre-computed center line through the colon, searching for polyps and other abnormalities. Fig. 1(a) illustrates a traditional interface for performing VC. The interface on the clearly shows the original CT slices in a scrollable stack, the 3D mesh representation of the colon and the main navigation interface for the virtual fly-through. Fig. 1(b) illustrates the rendering of a large polyp found by the radiologist. Compared to traditional OC, VC supports measurement and tracking tools, the ability to bookmark various points of interest inside the colon, and more importantly, a feature termed Electronic Biopsy [6]. Since VC visualization is driven by the CT data, it is possible to cast rays past the colon wall and accumulate the densities of the tissue underneath. Clinically significant polyps have been found to exhibit a distinct irregular high-density pattern and Electronic Biopsy utilizes a transfer function that maps high densities to a red color and low densities to a blue color to aid in the detection of that pattern. Fig. 1(c) illustrates the procedure applied directly from the user interface during the examination – the high-density irregular pattern is clearly visible in this case, indicating that the polyp is in fact an adenoma and should be removed surgically. This

Transcript of Immersive Virtual Colonoscopy - Stony Brook University · A. Virtual Colonoscopy Virtual...

Page 1: Immersive Virtual Colonoscopy - Stony Brook University · A. Virtual Colonoscopy Virtual Colonoscopy (VC) has been established as an alternative to optical colonoscopy (OC) for colon

Immersive Virtual Colonoscopy

Kaloian Petkov, Charilaos Papadopoulos, Student Member, IEEE, and Arie Kaufman, Fellow, IEEE Department of Computer Science and CEWIT

Stony Brook University Stony Brook, NY 11794-4400, USA

[kpetkov,cpapadopoulo,ari]@cs.stonybrook.edu

Abstract— Virtual Colonoscopy (VC) is an established non-invasive alternative to optical colonoscopy for colorectal cancer screening. Currently, radiologists examine VC data sets on a workstation computer using a single screen and simple navigation tools. The limited field of view increases examination time and introduces tedious interaction modalities. We propose conducting VC examinations inside an immersive visualization environment, such as the fully enclosed Immersive Cabin (IC) or semi-immersive workstation display configurations. The additional projection surfaces, life-size visuals and stereoscopic rendering in the IC provide the radiologist with a more intuitive navigation experience and allow for better utilization of the direct and peripheral vision to pick up visual cues related to the presence of polyps. As a result, the doctor is able to examine more of the colon surface at any given time, reducing total time for the screening and increasing the sensitivity of the screening procedure.

Keywords- Visualization, Virtual Reality, Scientific Visualization, Medical Imaging

I. INTRODUCTION

Colorectal cancer is a leading form of cancer in the US and results in more than half a million deaths worldwide every year. Existing screening techniques are very invasive, require a lengthy cleansing and as a result, compliance among patients above the age of 50 is low. Virtual colonoscopy [1,2] is a recently developed technique for non-invasive colon cancer screening which is much more comfortable for the patient and eliminates many of the health risks associated with the traditional approach. However, current implementations of Virtual Colonoscopy do not take advantage of recent developments in 3D and immersive display technologies. Immersive Virtual Colonoscopy improves the quality of the examination by allowing for improved shape perception due to the spatial cues produced by stereoscopic images. Also, the doctor is better able to examine surfaces that might otherwise be occluded simply by moving slightly inside the physical constraints of the immersive environment, or by using a very natural gesture-based interface.

We present an implementation of Virtual Colonoscopy in the Immersive Cabin (IC) [3], which is a fully-enclosed visualization environment designed to immerse the user in the visual data. The resulting system provides a more natural interface for interacting with the data and greatly enhances the field of view that the user experiences. In addition, stereoscopic

rendering allows for better shape detection based on visual depth cues, which is critical for finding polyps in the colon data. We also present a desktop system that offers many of the same advantages for medical examinations while greatly decreasing the significant financial burden associated with deploying the IC.

II. RELATED WORK

A. Virtual Colonoscopy

Virtual Colonoscopy (VC) has been established as an alternative to optical colonoscopy (OC) for colon cancer screening and shown to compare favorably [4,5]. It is non-invasive and does not require the traditional aggressive bowel cleansing of OC. The day before the procedure, the patient follows a low-residue diet and ingests a barium contrast liquid. A VC session involves the acquisition of a computed tomography (CT) scan of the patient’s abdomen. Due to the contrast material, it is possible to perform electronic cleansing on the CT data to remove any leftover stool from the CT slices. However, radiologists may instead elect to follow the traditional protocol for the physical colon cleansing. The colon surface is then extracted from the cleansed CT data using segmentation and it is visualized via volume rendering or other rendering techniques. The radiologist examines the virtual data by following the pre-computed center line through the colon, searching for polyps and other abnormalities. Fig. 1(a) illustrates a traditional interface for performing VC. The interface on the clearly shows the original CT slices in a scrollable stack, the 3D mesh representation of the colon and the main navigation interface for the virtual fly-through. Fig. 1(b) illustrates the rendering of a large polyp found by the radiologist.

Compared to traditional OC, VC supports measurement and tracking tools, the ability to bookmark various points of interest inside the colon, and more importantly, a feature termed Electronic Biopsy [6]. Since VC visualization is driven by the CT data, it is possible to cast rays past the colon wall and accumulate the densities of the tissue underneath. Clinically significant polyps have been found to exhibit a distinct irregular high-density pattern and Electronic Biopsy utilizes a transfer function that maps high densities to a red color and low densities to a blue color to aid in the detection of that pattern. Fig. 1(c) illustrates the procedure applied directly from the user interface during the examination – the high-density irregular pattern is clearly visible in this case, indicating that the polyp is in fact an adenoma and should be removed surgically. This

Page 2: Immersive Virtual Colonoscopy - Stony Brook University · A. Virtual Colonoscopy Virtual Colonoscopy (VC) has been established as an alternative to optical colonoscopy (OC) for colon

particular finding matches the result from the OC that was performed in the same session as the acquisition of the CT data.

B. The Immersive Cabin

A number of different solutions have been developed for exploring large datasets in an immersive fashion such as Head Mounted Displays (HMDs) and the CAVE (CAVE Automated Visualization Environment) [7]. Both provide much larger fields of vision, improved depth perception via stereoscopic rendering and a better understanding of the scale and spatial relations between the data. While HMDs allow for arbitrary views inside the virtual world, they present significant drawbacks related to the size of the device that the user needs to wear, the need for a wired connection to a host computer and the complete isolation from physical space that the user occupies. CAVEs provide a much more natural and unburdened visualization experience since they recreate the virtual environment via projections on wall and floor surfaces. This eliminates the need for a virtual avatar and allow for augmented reality applications. The first CAVE provided 3 back-back projected walls and a front-projected floor surface.

The Immersive Cabin (IC) is a later design based on the CAVE that contains 5 projection surfaces (4 walls and a floor) on which we render stereoscopic images. The visualization system contains 2 projectors and 1 workstation per screen and the rendering framework handles all image generation and synchronization tasks. In particular, we are able to display very dense meshes for the colon data, which are obtained from the original CT slices using the Marching Cubes method [8]. In addition, a high quality volume renderer is available for direct visualization of the CT data. The system also provides a convenient way for specifying the transfer functions and lighting parameters from within the IC using a small wireless tablet. The Immersive Cabin is depicted in Fig. 2.

III. IMMERSIVE VIRTUAL COLONOSCOPY IN THE IC

The IC rendering framework handles the projections on the 5 display surfaces automatically using a distributed scene graph

among the 5 rendering nodes in the IC GPU cluster. The user interacts with the head node of the cluster where the input is processed and updates to the main camera are streamed over a

(b) in the IC, images are back-projected onto the walls and front-projected onto the floor via a hanging mirror.

Figure 2. Outside and inside views of the Immersive Cabin

(a) Outside view of the Immersive Cabin near the entrance.

Figure 1. (a) Virtual Colonoscopy interface on the desktop; (b,c) polyp identification and examination during the exam.

(a) VC user interface (b) Polyp found during the exam (c) Electronic Biopsy tool

Page 3: Immersive Virtual Colonoscopy - Stony Brook University · A. Virtual Colonoscopy Virtual Colonoscopy (VC) has been established as an alternative to optical colonoscopy (OC) for colon

low-latency network. Each of the 5 rendering node then applies a transformation on the incoming data to create the final projection matrix. This process is transparent to the user of the system and automated for the developer of the VC application.

Interaction with the system is possible through a number of physical devices. The default scene navigation utilizes a gamepad or optionally a 3DConnexion Space Navigator that provides 6 degrees of freedom in a compact device. A wireless tablet can be used to access the main interface from within the IC and we have tested the interaction with the colon data using the Nokia N800 (4.13in screen) and the larger IBM Thinkpad X41 (12in screen). By default, the virtual camera follows the pre-computed center line in a similar fashion to the desktop navigation interface, although care has to be taken to minimize camera rolls that may disorient the user.

The volume rendering for the VC is an extension to the volume rendering module in the IC framework and it is tightly integrated with the scene graph. We have implemented a GPU-based single-pass ray-casting approach [9] with iso-surface extraction, a Cook-Terrance light transport model and distance-based light attenuation. The advanced lighting is used to help the examiner to better perceive shapes and distances in the cavern-like structures of the colon data. Fig. 3 illustrates the rendering technique for a section of the colon with a suspected cancerous polyp.

We can achieve interactive performance in the IC for 512×512×451 16-bit CT scan of a patient’s abdomen. Since the examination in VC consists of traversing the colon along the centerline, we rely heavily on optimizations such as empty space skipping and early ray termination, while additional computational resources are dedicated to high quality lighting and shading at the iso-surface representing the internal colon wall. Fig. 4 contains the final output of the GPU cluster for a single position and Fig. 5 shows a user performing the colon navigation in the IC using the volume rendering visualization module. Additionally, he may select to examine only the 3D mesh of the colon surface. This reduces the quality of the images in the IC, however it also increases the rendering speed

and provides a smoother animation even for an older generation GPU cluster.

We have also installed an 8-camera infrared tracking system from NaturalPoint, which allows us to track a number of objects inside the IC. The shutter glasses that the user wears to see the stereoscopic effect contain 5 passive reflective markers, which allow our framework to track the user’s head within the physical confines of the projection screens. This allows us to compute the correct perspective as the head moves within the IC and provides a very natural immersive examination of the colon surface. For example, the radiologist can take a physical step forward and look down and back to examine the back side of a haustral fold. In the non-immersive VC, this process requires traversals of the colon data in both the antegrade and the retrograde directions, increasing the overall examination time. In addition, we provide spherical markers to track the position of the user’s hands for a simple and intuitive gesture interface. For example, tools such as polyp measurement and the Electronic Biopsy can be activated with a flick of the hand while more complex motions can be used to control the navigation through the data.

IV. IMMERSIVE VIRTUAL COLONOSCOPY ON THE DESKTOP

The system for Immersive Virtual Colonoscopy presented so far offers a number of benefits – more natural interaction with the data, greatly enhanced field of view and better shape detection through the use of stereoscopic rendering. However, building an Immersive Cabin requires a significant financial investment and even a basic CAVE can cost tens of thousands of dollars. In addition, the projectors, the GPU cluster, and all the hardware for mounting the screens necessitate a large lab space, which may not be readily available in densely-instrumented medical centers.

Figure 4. Immersive rendering for a rectangular visualization environment. Individual tiles represent separate projection surfaces in the IC and are

rendered by individual GPU cluster workstations.

Figure 3. Rendering in the IC using volumetric VC CT data

Page 4: Immersive Virtual Colonoscopy - Stony Brook University · A. Virtual Colonoscopy Virtual Colonoscopy (VC) has been established as an alternative to optical colonoscopy (OC) for colon

As an alternative, a more easily accessible immersive environment can be constructed on the desktop for performing VC examinations. Our design consists of multiple monitors tiled at an angle to provide a visual environment that extends the field of view into the data. Depending on the requirements, a basic design with 3 monitors in a single row can be used to extend the horizontal viewing angle, while additional monitors can be mounted above and below to increase the vertical coverage. LCD panels capable of stereoscopic rendering are readily available in a variety of size and can be utilized to further enhance the immersion. Together with a headtracking system based on NaturalPoint’s TrackIR or Optitrack cameras, this installation provides many of the benefits of VC in the Immersive Cabin at a fraction of the cost.

Eliminating the GPU cluster greatly reduces the complexity of deploying an Immersive VC system. Although a small-scale cluster can be created cheaply and in a limited space, recent advances in graphics processors allow a single GPU to drive multiple screens in high definition resolutions, even when under an intense visualization workload. The ATI FirePro V9800 is a workstation-class graphics board that can drive 6 screens at a resolution of 2560×1600 while costing approximately $2000, effectively replacing multiple nodes of a GPU cluster. Up to 4 FirePro boards can be used in a single workstation together with a FirePro S400 synchronization module for genlock and framelock capabilities.

V. CONCLUSIONS AND FUTURE WORK

We have presented a novel method for conducting Virtual Colonoscopy, inside an immersive stereoscopic environment such as the IC. The examination process is much more intuitive, reduces the total examination time and can improve the polyp detection accuracy. For cases in which a full-size immersive visualization environment is not feasible, we propose examining VC data sets on a desktop computer with a number of screens in a partially immersive arrangement.

We plan on using a conformal transformation method for partially immersive visualization environments in order to regain visual information that is not available due to missing display surfaces. The advantage of using a transformation based on conformal mapping is that shapes are preserved locally and therefore the polyps in the data will retain their overall shape. We have completed a preliminary version of our conformal visualization system, and based on conversations with a radiologist, the results seem promising.

The wireless tracking system in the IC presents a number of unique opportunities for interactive applications. One of our major future goals is to develop a library of domain-specific gestures that the examining radiologist can use as a method of interaction for the system and its analysis tools, beyond the basic hand gestures described so far.

Our tracking system in the IC can also be used to obtain positional information for physical devices in the IC, in addition to providing gesture detection capabilities. One application involves the use of the wireless tablet as a window into additional information over the images shown in the IC. For example, we can show the output of the Electronic Biopsy directly on the tracked tablet, and the user selects the areas for the examination simply by moving the tablet in the IC and zooming can be achieved by holding the tablet closer or farther away. This can be a very intuitive user interface that utilizes the IC to provide the overall context for the Electronic Biopsy images which convey little information about the 3D shape of the observed polyps and the nearby tissue.

ACKNOWLEDGMENT

This work has been supported by NIH grant R01EB7530 and NSF grants IIS0916235, CCF0702699 and CNS-0959979. The VC datasets have been provided through the NIH, courtesy of Dr. Richard Choi, Walter Reed Army Medical Center.

REFERENCES [1] L. Hong, S. Muraki, A. Kaufman, D. Bartz, and T. He, "Virtual voyage:

interactive navigation in the human colon," ACM SIGGRAPH, 1997, pp. 27-34.

[2] C. D. Johnson and A. H. Dachman, "CT Colonography: The Next Colon Screening Examination?," Radiology, vol. 216, pp. 331-341, 2000.

[3] F. Qiu, B. Zhang, K. Petkov, L. Chong, A. Kaufman, K. Mueller and X. D. Gu, "Enclosed Five-Wall Immersive Cabin," Proceedings of the 4th International Symposium on Advances in Visual Computing, 2008, pp. 891-900.

[4] P. J. Pickhardt, et al., "Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults," The New England Journal of Medicine, vol. 349, pp. 2191-2200, 2003.

[5] W. Hong, J. Wang, F. Qiu, A. Kaufman, and J. Anderson, "Colonoscopy simulation," Proc. SPIE Medical Imaging, vol. 6511, 2007.

[6] A. E. Kaufman, S. Lakare, K. Kreeger, and I. Bitter, "Virtual colonoscopy," Communications of the ACM, vol. 48, pp. 37-41, 2005.

[7] C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. V. Kenyon, and J. C. Hart, "The CAVE: Audio Visual Experience Automatic Virtual Environment," Communications of the ACM, vol. 35, pp. 64-72, 1992.

[8] W. E. Lorensen and H. E. Cline, "Marching Cubes: A high resolution 3D surface construction algorithm," Computer Graphics (Proceedings of SIGGRAPH 1987), vol. 21, pp. 163-169, 1987.

[9] M. Hadwiger, J. M. Kniss, C. Rezk-salama, and D. Weiskopf, Real-time Volume Graphics: A K Peters, 2006.

Figure 5. Virtual Colonoscopy in the IC.