~ n v i r o n ~ ~ n ~ for Visualization and Sonification of Br tivr

rogress and availability of multimedia and virtual reality (VR) technology

make possible a new treiid of perceptual data presentation. As computer perfor- mance increases, the major bottleneck could be the human-computer interface. The bandwidth of this interface will be bound by characteristics of human per- ception, and hence the quest for a new pre- sentation paradigm has commenced in different scientific fields. Techniques de- veloped in VR facilitate multiple data-stream presentation and navigation through huge data sets. New iminersive environments are particularly appropriate to improve insight into complex biomedi- cal phenomena, which are naturally multi- dimensional.

As an extension to visualization, which gives predominantly spatial distri- bution, acoustic rendering may improve temporal cues. The technique of data pre- sentation using variable sound features is called sonification [ 1-41. In this article, we present early efforts toward perceptualization of biomedical data and introduce a novel method of multimodal data presentation with possible clinical and research applications. In particular, this environment for monitoring brain electrical activity consists of 3D visual- ization synchronized with data sonification of electroencephalogram and magnetoencephalogram (EEG/MEG) data. Visualization is based on topo- graphic maps projected on the scalp of a 3D head model. Sonification implements inodulations of natural sound patterns to reflect certain features of processed data and helps create a pleasant acoustic envi- ronment. This feature is particularly im- portant for prolonged system use.

~ v e r ~ i e w Efficient perceptualization of bio-

medical data requires a multidisciplinary approach, including the fields of com- puter science, computer engineering, psychology, and neurophysiology. The inherent nature and complexity of bio- medical data presents a fertile field where multimodal presentation finds its

50 0739-5175/99/$10.0001999 IEEE

Emil Jovanov1,2, Dusan Starcevic3, Vlada Radivojevic4, Aleksandar Samardzic5,

Vladimir Simeunovic2 'The ~ n i v e r s i ~ of A l n b a m ~ nt ~untsville

'Institute "Mihailo Pupin", 3e l~rade 'Fucu l~ of ~ ~ g a n ~ z a ~ i o n a l S~ien~es, Eelg~Qde

~lns~i tute of Mental Health, ~ e l g r Q d ~ 'School of Electrical E ~ ~ i n e e r i ~ g ,

Univers i~ of 3 e l ~ r a ~ e

place naturally. Availability of VR tech- nology moved the primary barrier to ef- fective data presentation to the user interface. One of the most important en- vironments for scientific exploration of virtual worlds is called the Computer Au- tomated Vision Environment (CAVE), as developed at the Electronic Visualiza- tion Lab of the University of Illinois, Chicago [ 5 ] . It is a multiperson VR sys- tem that projects computer-generated images onto three walls and the floor. Since 1992, it has became the de-facto standard for high-resolution, profes- sional VR environments. In the field of biomedical engineering, it has been effi- ciently applied to visualization of com- plex molecular structures [6-71 and to audio feedback for exploration o f volu- metric data [8].

New input technologies (such as eye gaze, foot-based input, and speech) and outputs (such as tactile) influence not only the VR environment, but also gradually change the desktop environment. Ever de- creasing costs of this technology resulted in its commercial availability for desktop computers. Multimodal presentations al- ready in use include visual sense, hearing, touch, and for some experiments, the sense of smell [9-lo].

Principally, there are two possible methods of multimodal data presenta- tion. The simplest is signaling of state transitions or indication of certain states, which is often implemented as an audio alarm, even on your PC. The sec- ond is presentation of current values in a data stream. Additional modes may be

IEEE ENGINEERING IN MEDICINE AND BIOLOGY

employed either as a redundant presentation emphasizing cert features, or to introduce new data chan- nels [ 1 11. Redundant presentation cre- ates artificial synesthetic perception of the observed phenomena [ 121. Artificial synesthesia (syrz = together plus azsthesis = perception in Greek) gener- ates sensory joining, in which information of one sense is a nied by a perception in another sense. Multisensory perception could improve understanding of complex phenomena by giving other clues or triggering dif- ferent associations. In addition, an acoustic channel could provide new iii- formation channels without information overloading.

Audio feedback for positional control could be a significant aid for surgeons in the operating room. Just as musicians use aural feedback to position their hands, surgeons could position instruments ac- cording to a preplanned trajectory, preplaced tags or cues, or anatomical models. In a U.S. Defense Advanced Re- search Projects Agency (DARPA) fi- nanced project, Computer Aided Surgery, Inc., (New York) [13], developed a surgi- cal training simulator in which multiple parallel voices provide independent chan- nels or positional information that are used as feedback during simulation of an actual operation [ 131.

Sonification of EEG sequences has been applied to detect short-time synchro- nization during cognitive events and per- ception (see [14] for a web site where you can listen to an example) Each electrode is assigned a different musical instrument digital interface (MIDI) instrument, and EEG synchronization is perceived as syn- chronous playing during data somfication.

There was also an attempt to use an au- dible display of patient respiratory data for anesthesiologists in the operating room [15]. It was found that anesthesiolo- gists spend only 7% to 20% of their intraoperative time observing patient monitors. Thus, audible display may in- crease the awareness of the patient's re- spiratory condition in the operating room.

Januory/February 1999

Sonification Early sonification applications mostly

used the so-called “orchestra paradigm,” where each data stream has an assigned instrument (flute, violin, etc.) [2, 81. Data values are then represented by notes of different pitch. The main advantage of this approach is the possibility of applying standard MIDI support, using the applica- tion programming interface (API) system. Unfortunately, this approach often leads to a cacophony of dissonant sounds, which makes it hard to discern prominent features of the observed data streams.

Multimodal data presentation is a complex problem, due to the nature of cognitive information processing [ 161. Efficiency of sonification, as an acoustic presentation modality, depends on other presentation modalities. The most impor- tant advantages of acoustic data presenta- tion are: . Faster processing than visual presen-

tation. . Easier to focus and localize attention in space, which is appropriate for sound alarms. . Good temporal resolution (almost an order of magnitude better than vi- sual). . An additional information channel, releasing the visual sense for other tasks . The possibility of presenting multi- ple data streams simultaneously

However, all modes of data presenta- tion are not perceptually acceptable. In applying sonification, one must be aware of the following difficulties of acoustic rendering:

Difficulty in perception of precise quantities and absolute values. . Limited spatial distribution. . Some sound parameters are not inde- pendent (pitch depends on loudness) . Interference with other sound sources (e.g., speech). . Absence of persistence. . Dependence on individual user per-

It can be seen that some characteristics of visual and audio perception are com- plementary. Therefore, sonification natu- rally extends visualization toward a more holistic presentation.

There are many possible ways of sonic presentation, so the system must provide the ability to extract the relevant diagnos- tic information features. The most impor- tant sound characteristics affected by sonification procedures are the following:

ception.

. Pitch is the subjective perception of frequency. For pure tones, it is a basic frequency, and for sounds it is deter- mined by the mean of all frequencies weighted by intensity. Logarithmic changes in frequency are perceived as linear pitch changes. Most people cannot estimate exact frequeincy of a sound (as for absolute pitch).

f m Timbre is a characteristic of the in- strument generating the sounds, which distinguishes it frorn other sounds of the same pitch and volume. The same tone played on different in- struments will be perceivedl differ- ently. Timbre could be used to represent multiple data streams using different instruments. . Loudness or subjective volume is proportional to physical sound inten-

. Location of a sound source may rep- resent information spatially. A sim- ple presentation modality may use the Balance of stereo sound to con- vey information.

Although mostly used as a csomple- mentary modality, sound could serve as a major human-computer interface modal- ity [17]. Aural renderings of pictures and visual scenes represent an important and promising extension of natural language processing for visually handicapped com- puter users. The same technology is di- rectly applicable to a wide range of hands-free, eyes-free computer systems.

sity.

Visualization and Sonification of Brain Electrical Activity

Visualization provides significant sup- port for the understanding of complex processes. One of the most important challenges for contemporary medicine and science is brain function. Possible in- sight into such functions could be facili- tated by visualization of the brain’s electromagnetic activity, observing either its electric component recorded1 on the scalp (EEG) or magnetic field in tlne vicin- ity of the head (MEG). The EEG lhas been routinely used as a diagnostic tool for de- cades. Recently, the MEG has been used in addition to complete the picture of un- derlying brain processes. The head is al- most transparent to magnetic fields, while its inhomogeneities (caused by fluids, skull, and skin) considerably influence EEG recordings.

Topographic maps of different param- eters of brain electrical activity h#ave been .commonly used in research andL clinical practice to present spatial distribution of

activity [18]. At first, topographic maps representing the activity on 2D scalp pro- jections were used, showing a represented static picture from the top view.

EEG brain topography is gradually be- coming a clinical tool. Its main indication is to aid in the determination of the pres- ence of brain tumors, other focal disease of the brain (including epilepsy, cerebrovascular disorders, and traumas), and disturbances of consciousness and vig- ilance (such as narcolepsy and other sleep disorders, grading the stages of anesthesia, evaluation of coma, and intraoperative monitoring of brain function). It is a valu- able tool in neuropsychopharmacology, studying the effects of drugs acting on the nervous system (hypnotic, psychoactive, antiepileptics, etc.). In psychiatry, EEG brain topography has been used to identify biological traits of certain disorders such as depression and schizophrenia, early onset of Alzheimer’s disease, hyperactivity with or without attention deficit disorders in children, autism, etc. [18-201. The EEG is also used as a therapeutic tool in various biofeedback techniques for the relief of tension headaches and stress disorders, and there are attempts to use it in more serious diseases such as epilepsy [21].

Recent advances in computer graphics and increased processing power have pro- vided the means of implementing 3D top- ographic maps with real-time animation. 3D visualization resolves one of the most important problems in topographic map- ping-projection of the scalp surface onto

January/Februory 1999 IEEE ENGINEERING IN MEDICINE AND BIOI.0GY 51

a plane; other problems of topographic mapping include: . Interpolation methods . Number and location of electrodes . Score to color mapping

While in CT and PET images every pixel represents an actual data value, brain topographic maps contain observed val- ues only at electrode positions. Conse- quently, all the other points must be spatially interpolated using known score values calculated from electrode posi- tions. Therefore, a larger number of elec- trodes makes more reliable topographic mapping possible. Electrode positioning is usually predefined (e.g. , the International 10-20 standard), although for some experiments, custom electrode positioning could be used to increase spa- tial resolution over certain brain regions. Finally, color representation of data val- ues may follow different paradigms, such as spectrum (using colors of visible spec- trum, from blue to red) and heat (from black through yellow to white) [IS].

Isochronous representation of ob- served processes preserves genuine pro- cess dynamics and facilitates perception of intrinsic spatio-temporal patterns of brain electrical activity. However, anima- tion speed depends on perceptual and computational issues. Commercially available computer platforms can create an animation rate in the order of tens of frames per second, depending of image size and score calculation complexity [22]. Although the animation rate can go up to 25 framedsec, the actual rate must be matched with information processing capabilities of a human observer. Other- wise, problems such as temporal summa- tion and visual masking may arise [23]. Both effects occur if the frame rate is too high and when details on adjacent maps interfere, which causes interference of de- tails on adjacent maps and creation of false perceptions.

The most important computational is- sues for real-time execution are complex- ity of score calculations, image size, and animation rate.

System Organization Our first visualization prototypes have

clearly shown the necessity for an experi- mental environment in which different perceptual features could be easily set and their diagnostic values explored. We de- veloped TEMPO (temporal EEG map- ping program) in Visual C++ for the

The program was developed to test user interfaces and the most important percep- tual features of visualization, including animation control and speed, evaluation of scores, color mapping (look-up tables), background properties, scene lighting, and model evaluation. A typical user in- terface of TEMPO is given in Fig. 1.

The TEMPO system consists of three parallel execution threads: data acquisi- tion, score calculation, and score visual- ization. They are synchronized by means of critical sections, and can be executed in parallel on different processors. Even in a single-processor system, data acquisition is usually supervised by an intelligent controller an intelligent controller on an A/D board, and score calculation can be performed by an add-on DSP processor, while the graphic coprocessor handles vi- sualization tasks.

EEG data can be retrieved either on-line (from the A/D converter board) or off-line (from a file). We implemented an input filter for standard EEG files gener- ated by RHYTHM 8.0 (Stellate Systems Westmount, Quebec) and now implement the ASTM EEG file format standard [24].

Sound processing relies on the Microsoft DirectX concept, intended to provide faster access to system resources and lower processing la tency. Directsound facilitates low-latency mix- ing and access to accelerated sound hard- ware, as well as real-time control of reproduction parameters. We developed software support for sonification as the DLL library (dxSonify.dll), which makes use of standard Directsound procedures. Custom procedures make possible the use

of natural sound patterns, stored as a stan- dard audio file. For instance, we have found sounds of a creek or bees as a natu- rally pleasant basis for sonification. The application modulates the pitch, volume, or balance of the selected sound pattern according to the values of the data stream to be sonified.

Experiment and Results We investigated the influence of cog-

nitive tasks on the spatio-temporal pattern of steady-state visual evoked potentials (SSVEP). In our experiment, the cogni- tive task was sustained attention during focused gaze as a simplest form of mental activity. We assume that changed SSVEP can indicate the different reactivities of brain regions to visual stimulation due to information processing. SSVEP was re- corded on healthy adult volunteers using strobe stimulation with variable flash fre- quency (3-20 Hz), changing in 1 Hz incre- ments. During each session, the EEG was recorded during a control state with strobe stimulation with eyes opened and relaxed, and during focused gaze. In the relaxed state, the gaze was fixed on an object on the wall approximately 2 m behind the strobe stimulator. The strobe stimulator was at gaze level, 20 cm in front of the subject. In the second part of the experi- ment, during focused gaze, the subject was concentrating with eyes focused 1 cm above the nasion (the bridge of the nose directly under the forehead). Stimulation at each frequency lasted for 20 sec. We present here results from one subject with the characteristic of reproducible change in alpha frequency activity (8-13 Hz).

Windows 95/NT operating system [22]. 1. TEMPO visualization environment.

52 IEEE ENGINEERING I N MEDICINE AND BIOLOGY January/February 1999

During relaxed gaze, the subject had a highly resonant response to flash, with peak spectral power corresponding to flash rate. During mental activity, there was a fixed spectral peak at 10 Hz for every flash rate (from 3 to 20 Hz) over the r ight hemisphere. Detai led neurophysiological analysis will be pub- lished in a follow-up paper.

EEGs were recorded in an electromag- netically shielded room on MEDELEC 1A97 EEG machine, (Medilog, BV, Nieuwkoop. The Netherlands). Bandpass filter limits were set at 0.5 and 30 Hz.

Ag/AgCl electrodes, with impedance less than 5 kQ were placed at 16 locations (F7, F8, T3, T4, T5, T6, Fpl, Fp2, F3, F4, C3, C4, P3, P4,01,02) according to the Inter- national 10-20 system, with their average as the reference. The EEG output was digi- tized with 12-bit precision at a sampling rate of 256 Hz per channel. Records were analyzed off-line on artifact-free segments.

We present the changes of the spatio-temporal pattern of electrocortical activity as a series of topographic maps. Figure 2 represents a typical response to strobe stimulation: a fairly symmetrical

I t=O t=250 ms t=500 ms t=750 ms I 2. Example of bilateral parieto-occipital activation during 10 Hz photic stimulation with relaxed gazing.

t=O t=250 ms t=500 ms t=750 ms

3. Maximum degree of asymmetry of cortical response during 10 Hz photic stimula- tion with relaxed gaze.

t=750 m s 1 t=O t=250 ms t=500 ms

4. Typical sequence of cortical response during 10 Hz photic stimulation with fo- cused gaze.

January/Februory 1999 IEEE ENGINEERING IN MEDICINE AND BIOLOGY

bilateral activation of the parieto-occipital cortex. In this example, the EEG shows an increase in spectral power in the alpha band, which varies slightly in time. An in- dex of symmetry ( IS) is calculated as:

IS = (P, - P,) / (P, + P,)

where P, and Pz represent power of sym- metrical EEG channels, such as, e.g., OI and 02.

Average IS of' total alpha power during relaxed gaze was 0.019 (standard devia- tion 0.051). Notice that this asymmetry of response always has its maximum over the left (dominant) hemisphere, as shown in Figs. 2 and 3. On the other hand, when the experimental conditions are changed (i.e. when there is a sustained attention with eye focusing), a quite opposite pic- ture can be seen, with strong asymmetry of cortical activation. Alpha power now reaches its maximum over the right (nondominant) hemisphere, with a much stronger index of asymmetry: IS=0.328 (standard deviation 0.059). Lateralization of evoked responses can be clearly seen in Fig. 4. This parameter can be visualized as a variable time series on a graph, as shown in Fig. 5. However, it will result in visual overload to the observer. It is very hard to trace spatio-temporal activity on topo- graphic maps in one window and a vari- able graph time series in the other. Therefore, we introduced a global lefdright hemisphere index of symmetry as a new sensory channel-the audio stream. This new channel makes possible global spatio-temporal supervision, with the added acoustic channel carrying com- posite information of overall activity, with good temporal resolution.

During relaxed gaze, IS indicates bilat- eral activation, while during mental activity, right hemsphere alpha activity dominates. This finding indicates different reactivities of brain regions to visual stimulation during information processing. We assume that those changes reflect an altered cortical state of perception due to internal evoked potentials (mentally activated response to flash), and we see them as a possible mea- sure of the spatial distribution of brain infor- mation processing.

Lessons Learned The first pleasant surprise from this ex-

periment was that the performance of commercially available systems was good enough to support real-time visualization for most calculated scores. Therefore, the major system bottleneck becomes human

53

perceptual bandwidth. As an example, on a PC PentiumPro 166 MHz machine, with 64 MB RAM/5 12 KB L2 cache and Win- dows NT operating system, animation speed for a 320 x 240 pixel 3D map is 10 framedsec. Hence, visualization of EEGMEG scores could be executed even on standard PC platforms. This is particu- larly important in a distributed environ- ment so as to allow multiple site evaluation of stored recordings.

Unfortunately, there are no obvious de- sign solutions to a given problem. It is very hard to find the most appropriate paradigm, or sound parameter mapping, for a given application. Therefore, it is advisable to evaluate different visualization and sonification methods and determine per- ceptually the most suitable presentation. Moreover, creation of user-specific tem- plates is highly advisable, as perception of audio-visual patterns is very personal.

Selection of scores for multimodal presentation is another delicate issue rely- ing on human perception. A score selected for acoustic rendering may be used either as a new informat ion channel (sonification of symmetry in addition to visualization of EEG power) or as a re- dundant channel of visualized informa- tion. In introducing additional channels, one should be careful to avoid informa- tion overload. Redundant multimodal

presentations offer the ability to choose a presentation modality for a given data stream, or to emphasize the temporal di- mension for selected stream.

Presentation of standard topographic maps should be synchronized with stan- dard polygraph EEG signal presentation to resolve artifacts. Artifacts such as eye blinking generate very strange changes on topographic maps, which are easy to re- solve in standard time series.

In our example, the sonification method proved valuable in dynamic fol- lowing of some parameters of brain elec- trical activity that would be hard to perceive otherwise. From Figs. 1-3, it is obvious that asymmetry of brain response to strobe stimulation was extremely dif- ferent in two experimental situations, but only if we observe channels pair by pair. If we try to follow the changes over hemi- spheres as a whole (eight pairs of chan- nels), the observer would experience visual overload and an additional sensory modality would be needed. We used sonification to present a composite pa- rameter of overall brain activity.

Another application of sonification could be long-term EEG monitoring in outpatient clinical practice or intensive care units [25]. During examination of long EEG records, the physician needs to sustain a high level of concentration,

0.1

0

-0.1

-0.2

-0.3

-0.4

Dynamic Total LefVRight Alpha Power Ratio , I I I I 1 1

Normal Gazing :]

-0.5 0 2 4 6 8 10 12 14

t[sI

5. Variation of total alpha power index of symmetry over left and right hemisphere dur- ing normal and focused gazing. Dominance of right hemisphere during focused gaze can be clearly seen. This parameter could be efficiently sonified as sound balance.

54 IEEE ENGINEERING IN MEDICINE AND BIOLOGY

sometimes for more than an hour. Mouot- onous repetition of visual information in- duces mental fatigue, so that some short or subtle changes in the EEG signal could be missed. Additional acoustic information could help in alerting the observer [26].

Finally, a new generation of program- ming environments significantly reduces implementation efforts by providing sup- port for the most frequently used functions. As an example, there is no need to support change of viewpoint in a VRML (vir- tual-reality modeling language)-based model, as it is supported directly by the VRML viewer. We are now developing a new version of the visualization and sonification environment based on Internet technologies (VRML and Java).

Conclusions Complex experiments require sophis-

ticated visualization of a large number of parameters. However, physiological char- acteristics of human perception limit the number of visualized parameters and their dynamics. Multimodal data presentation could increase throughput, introduce new I data streams, and improve temporal reso- lution. The future of biomedical data pre- sentation will probably be marked with multimodal data presentation and integra- tion of multiple diagnostic procedures. In this article we have presented sonification as a second presentation modality. During analysis, multimodal parameters must be , set to maximize the distance of changes in the perceptual domain. We implemented sonification in our 3D EEG visualization environment to render synesthetic exten- sion of a selected data channel or to intro- duce new composi te parameters . Sonification improved the ability to as- sess genuine dynamics of brain electrical activity and to perceive inherent spatio-temporal patterns of brain electri- cal activity.

Emil Jovanov received the Dipl. Ing. ( '84), M.S. ('88), and Ph.D. ( '93) degrees in electrical engineering from the University of Belgrade, Yugoslavia. He has been working as a research scientist at

the Mihajlo Pupin Institute in Belgrade since 1984. Since 1994 he has been an as- sistant professor at the School of Electri- cal Engineering, University of Belgrade. He is currently a visiting assistant profes- sor at the University of Alabama at

January/February 1999

Huntsville. His research interests include multimedia, computer architecture, bio- medical signal processing, and scientific visualization.

Dusan Starcevic re- ceived the Dipl. Ing. ('72) and M.S. ('75) de- grees in electrical engi- neering, and the Ph.D. (' 83) degree in informa- tion systems from the University of Belgrade, Yugoslavia. He is cur-

rently an associate professor at the Fac- ulty of Organizat ional Sciences, University of Belgrade. Since 1987 he has been a visiting professor at the School of Electrical Engineering, University of Bel- grade, teaching computer graphics. His main research interests include distrib- uted information systems, multimedia, and computer graphics.

Aleksandar B. Samardzic is a postgraduate stu- dent working toward his Ph.D. thesis at the School of Electrical En- gineering, University of Belgrade. His main re- search topic is computer graphics, especially ray

tracing and scientific visualization. He re- ceived his B.S. in electrical engineering from the School of Electrical Engi- neering, University of Montenegro, in 1993 and two independent master's: first in computer engineering from the School of Electrical Engineering, University of Belgrade, in 1996, and the second in com- puter science from the Faculty of Mathe- matics. University of Belgrade, in 1998.

Vlada Radivojevic re- ceived the Medical Doctor ( '81) and Neuropsychiatrist ('89) degrees in medicine from the University of Belgrade, Yugoslavia. He is currently head of the Department of

Epileptology and Clinical Neurophysiology at the Institute of Mental Health in Bel- grade. His main research interest is comput- erized brain electrical activity analysis, including application of artificial neural net- works and theory of nonlinear dynamics in

epileptology and in investigations of higher cortical functions.

Vladimir Simeunovic was born in 1'970 in Jagodina, Yugcdavia. He graduated from the Department of Auto- matic and Electronic Engineering, Faculty of Electrical Engineering, in Belgrade, in Septem-

ber 1994. He has been a research scientist in the Computer Systems Department at the Mihailo Pupin Institute in Belgrade since 1995. His major research iiiterests include object oriented programming, user interface design, and intemet soft- ware technologies.

Address for Correspondence: Dr. Emil Jovanov, University of Alab,ama at Huntsville, 213 EB, Huntsville, AL 35899. Tel: (256) 890-6632. Fax.: (256) 890-6803. http://eb-p5.eb.uah.edu/ -jovanov/. E-mail: jovanov@ece . uah.edu

References 1. Kramer G (Ed.): Auditory Display, Sonifcation, Audqication and Auditory Inter- faces, Reading, MA: Addison Wesley, 1994. 2. Madhyastha TM, Reed DA.: Data sonification: Do you see what I hear?, IEEE Soft- ware, 12(2): 45-56, 1995. 3. Minghim R, Forrest AR: An illustrated analy- sis of sonification for scientific visualization, Proc. 6th IEEE Visualization Conference, VI- SUAL 95, IEEE Computer Societ:y Press,

4. Grinstein G, Ward MO: Introduction to Visu- alization: '96 Tutorial #2, Data Visualization, http://www.cs.uml.edu/-grinstei/tut/v96~tut2.ht ml. 5. Crnz-Neira C, Sandin DJ, DeFanti TA, Kenyon RV, and Hart JC: The CAVE: Audio visual experience automatic virtual environment, Comm. ACM, 35(6): 65-72, June 1992. 6. Akkiraju N, Edelsbrunner H, Fu P, Qian J: Viewing geometric protein structures from inside a CAVE. IEEE Computer Graphics and Applica- tions, 16(4): 58-61, July 1996. 7. Cruz-Neira C, DeFanti TA, Stevens R. and Bash P: Integrating virtual reality and high per- formance computing and communications for real-time molecular modeling. Proc. High Perfor- mance Computing '95, April 1995 8. Brady R, Bargar R, Choi I, Reitzer J: Audi- tory brcad crumbs for navigating volum(?tric data. IEEE Visualization '96, San Francisco, 1996. 9. Begault DR: 30 Soundfor Virtual Reality and Multimedia, Boston: Academic Press, Inc. 1994.

110-1 17, 1995.

10. Burdea GC: Force and Touch Feedback for Virtual Reality, New York: John Wiley & Sons, Inc,. 1996. 11. Edwards ADN, Holland S (Eds.) : Multime- dia Interface Design in Education. NATO AS1 Se- ries, Vol. F76, Springer-Verlag Berlin, 1992. 12. Cytowic RE: Synesthesia: phenomenology and neuropsychology a review of current knowl- edge. Sydney: Psyche, 2(10). 1995. 13. Wegner K, Karron D: Surgical navigation using audio feedback, in Morgan KS et a1 (Eds): Medicine Meets Virtual Reality: Global Healthcare Grid, Washington, D.C.: IOS Press, Ohmsha, pp 450-458, 1997. 14. Mayer-Kress GJ: Dynamicalresonances and synchronization of auditory stimuli and evoked responses in multi channel EEG, Proc. of Int. Conf on Auditory Display, ICAD'94; also http://www .ccsr.uiuc.edu/People/gmk/Pro- jectsEEGSound/ 15. Noah S, Loeb RG, Watt R: Audible Display of Patient Respiratory Data, Technical Report, University of Arizona, 1996. 16. Barnard PJ, May J (Eds.): Computers, com- munication and usability: Design issues, research and methods for integrated services. North Hol- land Series in Telecommunication, Amsterdam: Elsevier, 1993. 17. Raman TV: Auditofy User Interfaces: To- ward the Speuking Computer; Boston, MA: Kluwer Academic Publishers, 1997. 18. Duffy FH (Ed.): Topographic Mapping ofBrain Electrical Activity. Butterworth Publishers, 1986. 19. Lopes da Silva IFH: A critical review of clini- cal applications of topographic mapping of brain potentials, J. Clin. Neurophysiol., (7)4: 535-551, 1990. 20. Tsoi AC, So DSC, Sergejew A: Classifica- tion of electroencephalogram using artificial neu- ral networks, Advances in Neural Information Processing Systems, 6: 1151-1158, 1994. 21. Rockstroh B, Elbert T, Birmbauer N, Wolf P, Duchting-Roth A, e t al.: Cort ical self-regulation in patients with epilepsies, Epi- lepsy Research, 14: 63-72, 1993 22. Samardzic A, Jovanov E, Starcevic D: 3D visualisation of brain electrical activity. Proc. 18th Annual Int. Con$ IEEE/EMBS, Amsterdam, October 1996. 23. Klymenko V, Coggins JM: Visual informa- tion processing of computed topographic electri- cal activity brain maps, J. Clin. Neurophysiol., 1990,7(4): 484-497. 24. Standard Specification for E1467-94 Trans- ferring Digital Neurophysiological Data Between Independent Computer Systems, American Soci- ety for Testing and Materials, 1997. 25. van Gils M, Rosenfalck A, White S, Prior P, Gade J, et al.: Signal processing in prolonged EEG recordings during intensive care, IEEE EMB Maga- zine, 16(6): 56-63, November-December 1997. 26. Jovanov E, Starcevic D, Wegner K, Karron DB, Radivojevic, V; Acoustic rendering as sup- port for sustained attention during biomedical procedures, Proc. 5th Int. Conf on Auditory Dis- play, ICAD'98, Glasgow, UK, 1998.

Januory/Februory 1999 IEEE ENGINEERING IN MEDICINE AND B10L8DGY 55