Mario J Lorenzo CISD 792 10/5/2018

A Review Of

“Interactive visualization of volumetric data with WebGL in real-time”

Problem

In their paper “Interactive visualization of volumetric data with WebGL in real-time”, Congote et al focus on the problem of real-time rendering of volumetric data, such as medical images or weather radar, through the web-browser. The authors demonstrate the feasibility of conducting client-side volume rendering with WebGL. They discuss the challenges that affect performance, scalability, accuracy, and security with their approach. They envision the web-browser as replacing the native desktop applications used today for these visualization tasks.

Congote et al, observe that much of the rendering done through the web-browser involves polygonal meshes. They point out that real-time rendering of polygon models is straightforward given the embedded support by most graphics accelerators for raster render algorithms. The authors observe a deficiency for supporting volumetric datasets that are irregular, represented as scalar or vector fields. They assert that surface-based rendering techniques are not suitable for visualizing these kinds of images, but instead require Volume Rendering algorithms not available for the Web. The paper methodically outlines a Volume ray-casting approach to solve address this problem. Background and Prior Research

The scientific fields of Medical Imaging and Meteorology have been pioneer and

driving force behind active research in the areas of 3D visualization. Today, tools that provide visualization for medical scans such as CT and MRI or forecasting models that rely on Doppler radar serve as the primary tool for radiologists and meteorologist. Both of these professions base their decisions almost solely on their interpretation of these images. For Radiologist, the advent of CT and MRI scans provides data that can be rendered into a 3D form allowing a radiologist to observe the anatomical structure of the body and identify abnormalities that assist in their diagnosis of a patient. A Meteorologist interprets the results of radar scanning to understand the magnitude and trajectory of a storm. Both of these professions make critical decisions that impact human lives. Therefore, improvements to their visualization, such as improved fidelity and dimensionality, can improve the accuracy of the their decisions. Furthermore, the democratization of access to modern medicine through telemedicine has allowed experts to consult and collaborate globally with their patients through web-platforms (Min et al, 2018). The paper by Congote et al, provides a technical solution for bringing 3D medical (and radar imaging) visualization to the web.

Prior to WebGL, 3D visualization was performed through native applications or on the Web through 3rd party plug-ins that have to be installed in the browser and given

access to the local hardware, such as GPU, for rendering. Most often, these 3D images rendered by 3rd party plug-ins were based entirely on polygons meshes, which the authors assert are straightforward and handled by raster rendering algorithms implemented in most graphics accelerating hardware (i.e. GPU).

The approach taken by Congote et al relies on the early pioneering medical imaging research by Levoy on volume rendering where Levoy demonstrated that volume-rendering techniques produced better images from volumetric data than using conventional geometric primitives (Levoy, 1988).

Congote et al also leverage the work by Hadwiger et al on volume ray-casting. Hadwiger et al explain that volume ray-casting is the most basic, but most flexible volume rendering algorithm. They discuss ray-casting as a numerical method for evaluating the volume rendering integral. The volume rendering integral is based on integrating light interaction effects along a viewing ray on the optical model (Hadwiger, 2009).

Congote et al, provide a solution for rendering volumes through a web-browser using a variation of the original Volume Ray-casting algorithm adapted for the Web.

Significance

After conducting a brief literature review, it appears that Congote et al may have provided the first client-side Web-based volume rendering implementation. Their work proved the feasibility of developing medical imaging applications that can perform rendering on the client-side within a reasonable about of time without relying on a remote-server for computation. Because the rendering is performed locally at the client, this provides the ability to interact with the rendering in real-time. The ability to interact with a 3D image in real-time is a critical requirement for radiology visualization tasks (Haehn, 2014). By distributing the computation of rendering to each client, the authors assert the benefits of a scalable approach that is not limited by the number of users interacting with the images.

Congote et al implements their work using the WebGL standard and makes it available as a demo at http://demos.vicomtech.org/volren/. They consider this demo a side-effect of their work, but serves as a proof-of-concept that client-side web-based visualization works.

This paper does an excellent job in explaining the methodology and approach taken for adapting volume rendering using Ray-casting for WebGL. The paper also demonstrates the important applications of volume rendering to fields such as medical imaging and meteorology and the improved accessibility provided by the Web for these visualization tools. Congote et al show an example of taking a dataset of that represents a medical CT scan in the form of slices as shown in figure 1. They perform a pre-processing step where each image of the slice is arranged in a matrix configuration and stored as a single image. After applying their volume-rendering algorithm, they can generate a projected 2D image rendered from the perspective of a virtual camera. This projection approach is called the pin-hole camera, which project a 3D point onto a 2D point. Figure 2 shows an example of this rendering.

A critical aspect of rendering the image involves applying color to each pixel. An imaginary cube encapsulates the image and a projected ray through the cube that visits interacting pixels of that image. The paper describes the approximation of a tri-linear interpolation along with shade produced by illumination conditions determine the final color. The color will vary depending on its axis distance and other factors such as opacity and lighting. The paper notes the key formulations used for computing the components for the transfer function.

Further Research There appears to be ample additional research opportunity in this space. Congote et al listed several challenges that they faced with their approach that require additional work to improve. One of these challenges included latency encountered during the transfer of these large image files. Because each image slice is sent individually, the compression is done per image file. An interesting idea presented, and worth exploring, is to leverage video compression approach that eliminates temporal redundancy occurring between consecutive images in a sequence. Another areas of additional research that can improve the overall quality of the image being rendered is the inclusion of surface rendering within the volume rendering to provide a sense of surface texture that would help a radiologist better distinguish between tissues. Since the publication of this paper, ThreeJS framework has provided an abstraction over WebGL. An exploration of whether Three JS can help support these volume-rendering applications is worthy of investigation. An initial review found a couple of prototypes called XTK (X Toolkit) and ATM (A Toolkit for Medical Imaging) that were based on ThreeJS but did not appear to have the realism produced by Congote et al.

Figure2(Congoteetal)Figure1(Congoteetal)

Another challenge encountered during their work involves reducing the resolution of the medical image to accommodate the device (i.e. mobile, tablet, laptop etc) used for rendering and visualization. The paper explains that there is a mismatch between the resolution of the original image slices and the resolution of a laptop (4096x4096) and handheld (1024x1024). Medical images also use a sample bit depth that is bigger than the commonly supported 8 bits per sample. These issues were corrected as a pre-processing step before invoking their volume-rendering function. Additional work to dynamically handle and reduce image resolution on the client-side could reduce the amount of pre-processed images that have to be maintained by a server.

The authors cite the need for further optimizations in the implementation of the shaders used with the goal of improving performance. The authors also noted performance delays caused by the browsers implementation of Javascript such as the setTimeout() function having a 10ms average overhead per call. This could have significant performance impacts if there are a large number of computations that take less than 10ms.

Lastly, the authors discuss the use of additional techniques that can improve the render quality. The use of interactive and complex lighting is mentioned as a promising direction. The effect of lighting or the ability to control the light source and color may prove to be very helpful to an expert examining a medical image to identify an abnormality that is obfuscated under certain lighting but could be revealed if those controls were available.

Overall I found this paper to be very interesting and inspiring. The paper not only served as a source of knowledge for 3D graphics, specifically for the area of volume rendering and ray-casting, but also piqued my curiosity about the field of medical imaging and how computer graphics plays a vital role in advancing that domain.

References Reviewed paper: Congote, J., Segura, A., Kabongo, L., Moreno, A., Posada, J., & Ruiz, O. (2011, June). Interactive visualization of volumetric data with webgl in real-time. In Proceedings of the 16th International Conference on 3D Web Technology (pp. 137-146). ACM. Congote, J., Segura, A., Kabongo, L., Moreno, A., Posada, J., & Ruiz, O. – Demo of Real-Time Interactive Visualization of Volumetric Data with WebGL. http://demos.vicomtech.org/volren/.

Additional papers HADWIGER, M., LJUNG, P., SALAMA, C. R., AND ROPINSKI, T. 2009. Advanced illumination techniques for gpu-based volume raycasting. In ACM SIGGRAPH 2009 Courses, ACM, 1–166.

Haehn, D., Rannou, N., Ahtam, B., Grant, E., & Pienaar, R. (2014). Neuroimaging in the browser using the X Toolkit. Frontiers in Neuroinformatics, 101. LEVOY, M. 1988. Display of surfaces from volume data. IEEE Comput. Graph. Appl. 8, 3, 29–37.

Min, Q., Wang, Z., & Liu, N. (2018). An Evaluation of HTML5 and WebGL for Medical Imaging Applications. Journal of healthcare engineering, 2018.