Powervis: Empowering the User With a Multi-Modal Visualization System

R. Minghim rminghim@ icmsc.sc.usp.br

M.C.F. de Oliveira cristina @ icmsc. sc .usp .br

Department of Computer Science Institute of Mathematical Sciences of S5o Carlos

University of S5o Paulo CP 668 - S2o Carlos - SP - Brazil

Abstract

Visualization techniques provide valuable tools for interpreting the huge amount of data being generated by new detection devices and simulation algorithms. Their effectiveness is limited, among other reasons, by the lack of a comprehensive approach towards considering human needs and exploring their abilities. This paper proposes a system and a model f o r integrating visualization and imaging techniques in order to study and implement strategies to enhance data and phenomena interpretation. This system (PowerVis) aims at providing new forms of graphical and auditory mappings that improve conventional approaches to visualization systems.

1 Introduction

The aim in Scientific Visualization (ViSC) is to render images from a data set (typically obtained from detection devices or numerical simulations) which are to be interpreted by users who wisb to gain insight into the meaning conveyed by the data. Despite their limitations, currently available visualization tools provide invaluable help to users that would otherwise be lost in a sea of numbers and measurements. Through a suitable interface,

they can interact with the data set and with the graphics, producing new images which may reveal new structures and relationships in the data and promote understanding of phenomena. This process should be supported by techniques from the field of Computer Vision (CV), whose aim is to analyze an image in order to detect features of interest automatically.

Although rendering and recognition of images are often seen as opposite processes, somewhat linked by image processing, we may say that in CV, as in ViSC, the ultimate goal is to understand what is conveyed by the image. Several authors [l-71 propose a more integrated view of the fields. There can be found many common aspects among the techniques used to visualize, quantify and interact with data and models, as well as important common applications. In this context, it is widely accepted that human Perception is an important issue both for ViSC and CV [%lo]. In ViSC, inadequate graphical mappings may result in misleading pictures. This problem is partly caused by the fact that, to date, very few visualization techniques approach the problem of considering the way humans perceive and understand graphics.

The combination of imaging and graphics techniques for gaining interpretation has been called visual computing [7]. Its principles are meant to be applied in the development of PowerVis, a multi-modal visualization system aimed at providing users with comfortable,

106 0-8186-8058-X/97 $10.00 0 1997 IEEE

meaningful and natural functions to correctly and easily map their data not only to graphics, but also to sound. In PowerVis, this is being done by considering aspects of perception and task analysis for further exploring and guiding the user through data set features. Aspects of automatic feature detection are being considered both in pre-processing the data to support visual (and aural) attribute mapping, and to identify regions of interest in the picture resulting from the graphical process. Sound has recently gained much attention in the scientific community as a powerful tool, not only to accompany pictures, but also to convey meaningful information [ 1 1 - 131. Its use can greatly enhance the interpretation of visually overloaded information spaces and support information gathering activities in many situations. PowerVis intends to meet its goals by reviewing conventional visualization techniques in the light of perceptual and cognitive issues and the ways we handle information gathering by use of visual and aural stimuli.

2 PowerVis The PowerVis system consists of severa! modules as

shown in figure 1. It takes conventional visualization techniques and systems plus feature extraction and creates a layer of code that includes changes in the algorithms, post-processing of the graphical functions, and additional

algorithms to provide perceptually effective visual and aural representations. The choice of visual mappings is to be supported by a rule-based module for guiding the user. The development of the sound mappings represents an extension of work being developed for a number of years by one of the authors [12,14,15]. PowerVis is being implemented through the inclusion of additional visualization modules and techniques into the VTK (Visualization Toolkit) visualization system [ 161.

2.1 3D Visualization and Feature Extraction Module

Most visualization systems offer the users a wide range of choices of techniques as well as controls for each technique. However, leaving those controls completely to user choice poses serious difficulties to the task of adjusting visual appearance of the resulting graphics (which are likely to hinder its interpretation power) [17- 191. It is particularly important to consider domain- specific characteristics in generating a proper data visualization. For example, professionals in medical and biological sciences are used to visualizing gray-scale images; users of Geographical Information Systems tend to adopt a standard color-coding that maps sky and sea to blue, brown to earth, and so on. Our aim is to identify such

,-d 3D Visualization 14

'igure 1. PowerVis Basic Architecture.

107

RANGES I 1

SCALAR

Figura 2. PowerVis Scheme for Implementation of Visualization Rules. relevant domain-specific knowledge for particular domains, and incorporate such knowledge within our rules. The V3D module (shown in figure 1) is to include a rule-based system that supports the choice of the visual attributes (such as color, shape and texture) in graphical mappings for visualization purposes. We present an extension to the work developed by Bergman et al. [18], who propose rules for color map selection associated with a previously defined architecture for rule-based visualization [ 161. Our extension includes aspects of domain of application, as well as shape perception, not treated in their work.

Figure 2 identifies the basic classes of rules being implemented, related to color, shape and domain. The top- right table in the figure shows the organization of the rules designed by Bergman et al. In their definition, they consider Spatial Frequency trend (classified as Low or High) as well as the nature of the visualization tasks (isomorphic, segmentation and highlighting). The two sets of rules in the table represent the type of scalar data being mapped (ratio or interval). They use the HSL (Hue, Saturation, Luminance) model for color specification. For example, the rule for ratio type of data with low spatial frequency, for an isomorphic detection task, uses a color table with uniform Luminance, opponent Hue pairs and monotonically increasing Saturation from gray.

Our extension considers scalar data mapped to shape or to color, as well as two scalar fields, one represented by shape and the other by color attributes. Also, we add two new modules, one that includes basic rules for application domain (named Domain Rules in figure 2) and another that improves attribute mapping to account for non- uniform distribution of frequencies (Frequency Distribution). In our attribute selection there are similar tables for both shape and color manipulation in the case of a single scalar field represented by shape. Rules regarding isomorphic tasks decide whether and how to smooth the resulting isosurface, while rules regarding segmentation and highlighting tasks decide pseudo-illumination parameters (transparency, material, etc.) for individual isosurfaces, based on visual perception aspects of the parameters. In the case of two scalar fields represented simultaneously, one in surface and one in color, there is no distinction among the associated visualization tasks for the field presented by the isosurface. Therefore the rules only decide the type of smoothing of the surface, resulting in a simplified table.

Further research and a close interaction between system developers and visualization users are essential for the identification of additional cognition and perception aspects relevant to the visualization processes, and consequent implementation of additional rules.

108

2.2 Domain Test Cases

As am illustration, we now briefly describe applications in which we are particularly interested: the visualization of fluid flows and environmental data.

There are several difficulties involved in visualizing volumetric information and fluid dynamics provides a fine example of some of these. Visualizations of fluid position and movement are easy to obtain using conventional graphics, by showing the boundaries of the fluid in the container, as illustrated in figure 3. These pictures represent the results of a simulation of free surface flows yielding time-varying information of fluid [20,2 I]. Surface graphics in connection with animation are adequate for understanding movement. However, in this case the original data set produced by the simulation program includes data of higher dimensions, comprising at least two scalar fields (pressure and temperature) as well as a vector field (velocity). Pressure and temperature were originally visualized using isosurfaces and isocontours, and velocity was visualized using the hedgehog technique [20]. Those are useful visualizations for 2D simulations, but visualizing such data for the 3D simulation is a task that requires careful design of visual attributes. For these visualizations it is necessary either to reduce dimension of the display (as shown in [20]), or to find ways of combining different visual parameters to represent data and relationships from a complex set. A major goal in PowerVis is to use this type of configuration (volumetric organization with multiple data fields) as a platform for testing the visual and auditory rules designed.

Environmental data sets provide a second test case for the application and verification of theories on color use and representation by means of color, as previous work has confirmed [15]. Data from an environmental survey (maritime gas measurements) were collected and preprocessed to interpolate to a regular grid. Isosurface visualization was generated from the resulting grid. The goal was to analyze behavior of substances in order to detect temporal and positional changes in their distribution on the Noirth Sea. The first images from the data had proved themselves inadequate for interpretation due to poor choice of color tables. The users employed those versions of ‘bad’ color mappings and tried to change them until obtaining suitable ones. The need for greater flexibility was identified in changing the tables, as well as in viewing the results. It was noted that when the user is

concerned with individual values as well as with general trends in the data, contrast is essential. However, the placement of lower and higher values in the color scale presents a clear subjective component that must be taken into consideration. This reinforces existing guidelines of color coding for visual rules and influenced the rules as shown in figure 2.

b) Figure 3. Visualization of Free-Surface Flows: a) Cavity Filling Simulation. b) Splashing Drop Simulation.

This experiment also he detection and p~ovict part of the 3D v

processing is meant to identify patterns over one dimension, that is, in the direction of any of the axis, inside the volume. In PowerVis, conventional image analysis tools will be used to check temporal coherence of the information presented using surface plots. Results of this detection will be visualized and sonified. Future extensions will consider other forms of volume rendering. Visualizations for both applications are being developed without employing the rule-based architecture and will be compared with forthcoming results from the rule-based visualizations.

2.3 Data Sonification In the system ( S o u n d ) defined and implemented by

Minghim and Forrest [12,14], there are mappings to support direct data sonification, sonification of shape information and sonic investigation of volumetric content

109

Figure 4. Data-Flow Diagrams of the Sonic/Graphical Mapping and the Sonic Mapping Modules of Powervis.

(Volume Scan). S o u n d is being migrated, at the time of writing, to PowerVis, in order to extend its flexibility, by adding new sonification functions. Figure 4 shows sonification functions associated with the graphical mappings (they implement part of the functions of the modules SDAD, SVOL and IGS in figure 1). Sonifications are designed to accompany visual information, either as a complement or adding dimension to the display, and to work independently of visual feedback (for monitoring and eventual mappings). Because hearing is a sense less powerful than vision, in sonification most of the mappings for use in data exploration primarily considered how humans process sound streams and how sound interacts with the visual sense. Some of the aspects considered included sound property identification and distinction, type of hearing (musical or eventual), sound grouping and auditory scene analysis [ 12,221. Also, establishing correspondence between graphical and aural dimensions was fundamental (see figure 5) . Designed within these premises, sound signals are useful to reflect data relations and structures, and to provide adequate redundancy for confirmation of the graphics. Sound functions were designed to help relieve the visual channel, to add dimension, and to resolve ambiguity in the graphics.

In SSound, several properties were ‘sonified’ (content, value, normals, occupation, exceptions, curvatures, coordinates) and those are being extended to

provide sound mappings to many other properties (e.g. direction, position, range, layering, and concentration of values).

Figure 5 illustrates the technique developed for the mapping of graphical data to sound, and provides a first step towards development of rule-based sonifications. Before that can be done, however, further tests on the effectiveness of sound for specific tasks have to be performed. Initial tests of the sonifications confirmed sound utility for support to graphical presentation and debugging [14], and effective support to data analysis was also a result reported [ 151.

3 Conclusions

Work on visualization has demonstrated the need for better strategies in deciding how to interpret data with support of the computer. The PowerVis project means to pave the way to research in more natural ways of communication that best explore the potential of human perceptual abilities. The development of PowerVis feeds the discussion on integration of automatic detection of features and highly-perceptual multimodal tools to effectively assist in the interpretation of information. The proposed structure is being implemented incrementally, and extends previous initiatives in providing systems for perceptually effective data mappings to graphics and

110

sound. All the techniques developed can be extended to most domains that make use of visual interpretation of information.

/m ,-, KEF

PRJPERTY CALCULATION

00-0 SEOUENCE OFSOUNDS

[-L1 DATA PRORRTY

‘inure 5. Coordination and Svnchronization c 6und Streams to Graphical Ehtities.

References

I. Carlbom, I. Chakravarty and W.M. Hsu. “Integrating Computer Graphics, Computer Vision and Image Processing in Scientific Applications”. ACM SIGGRAPH Computer Graphics, Vol. 26, n. 1, pp. 8-16, January 1992. A.R. Forrest. “Rendering Versus Recognition” (Invited Talk). Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI’95), SBo Carlos, SP, Brazil, October 1995. I. Carlbom. “Visualization: Computer Vision Meets Computer Graphics”. Proceedings of the IEEE Workshop on Visualization and Machine Vision, IEEE CS Press, pp.

D. Silver and N.J. Zabusky. “Scientific Visualization and Computer Vision”. Proceedings of the IEEE Workshop on Visualization and Machine Vision, IEEE CS Press, pp. 55- 61, 1994. B.L. Yeo and S.P. Liou. “From Visualization to Perceptual Organization”. Proceedings of the IEEE Workshop on Visualization and Machine Vision, IEEE CS Press, pp. 62- 73, 1994. T.S. Newman. “Issues and Future Directions for Medical Applications of Computer Vision and Data Visualization” (Panel). Proceedings of the IEEE Workshop on Visualization and Machine Vision, IEEE CS Press, pp. 106,1994.

107-108, 1994.

[7] M. GroR. Visual Computing, Springer-Verlag, 1994. [8] B.E. Rogowitz and L.A. Treinish. “Data Structures and

Perceptual Structures”. Proceedings of SPIE Conference on Human Vision, Visual Processing and Digital Display, Vol. 1913, pp. 600-612, February 1993. J. Lansdown. “Visual Perception and Computer Graphics”. R.A. Earnshaw (ed.): Fundamental Algorithms for Computer Graphics, Springer-Verlag, pp. 1005-1026, 1991.

[lo] L.H. Reuter, P. Tukey, L.T. Maloney, J.R. Pani and S. Smith. “Human Perception and Visualization” (Panel). Proceedings of the IEEE Visualization’YO, pp.401-406, 1990.

[ 1 11 G. Kramer (ed.). Auditory Display: Sonifcation, Audifcation and Auditoly Intefaces, Addison-Wesley, 1994.

[12] R. Minghim. On Sound Support for Visualisation. PhD Thesis, School of Information Systems, University of East Anglia, UK, March 1995.

[13] T.M. Madhyastha and D.A. Reed. “Data Sonification: Do You Hear What I See?’. IEEE Sofhvare, Vol. 12, n. 2, pp. 45-56, March 1995.

[ 141 R. Minghim and A.R. Forrest. “An Illustrated Analysis of Sonification for Scientific Visualization”. Proceedings of the IEEE Visualization’95, IEEE CS Press, pp. 110-1 17, 1995.

[15] A.M. Day, R. Minghim, D.B. Amold and S. Tumer. “Visualisation and Sonification of Marine Survey Data”. R. Eamshaw et. a1 (eds.): Visualization and Modelling, Academic Press Inc., London, 1997 (to appear).

[I61 W.J. Schriieder, K. Martin and W. Lorensen. The Visualization Toolkit - An Object-Oriented Approach to 3 0 Graphics, Prentice-Hall, 1996.

[17] B.E. Rogowitz and L.A. Treinish. “An Architecture for Rule Based Visualization”. Proceedings of the IEEE Visualization’Y3, IEEE CS Press, pp.236-243, 1993.

[18] L.D. Bergman, B.E. Rogowitz and L. Treinish. “A Rule- based Tool for Assisting Colormap Selection”. Proceedings of the IEEE Visualization ’95, IEEE CS Press, pp. 118-125, 1995.

[19] H. Senay and E. Ignatius. “A Knowledge-Based System for Visualization Design”. IEEE Computer Graphics and Applications, Vol. 14 no. 6, pp. 36-47, November 1994.

[20] R. Minghim and M.C.F. de Oliveira. “Three-Dimensional Visualization of Free Sur$ace Flows”, Technical Note,

[21] M.F. Tome, A. Castelo Fo., J. Murakami, J.A. Cuminato, R. Minghim and M.C.F. de Oliveira. “Numerical Simulation of Axysimmetric Free Su face Flows”, Technical Note n. 030/96, ICMSC-USP, 1996.

[22] S. McAdams, S. and E. Biggand (ed.). Thinking in Sound - The Cognitive Psychology of Human Audition, Oxford University Press, 1993.

[9]

ICMSC-USP, April 1997.

111