In S. Göbel, U. Spierling, A. Hoffmann et al. (Eds.): Technologies for Interactive Digital Storytelling and Entertainment: Second International Conference, TIDSE 2004, Darmstadt, Germany, June 24 - 26, 2004, Proceedings. Lecture Notes in Computer Science Vol. 3105 / 2004, Springer-Verlag Heidelberg, pp. 119 - 125

Springer-Verlag

Learning from the Movie Industry: Adapting Production Processes for Storytelling in VR

Richard Wages1, Benno Grützmacher1 and Stefan Conrad2

1 Laboratory for Mixed Realities, Schaafenstr. 25, D - 50676 Cologne, Germany {wages, gruetzmacher}@lmr.khm.de

2 Fraunhofer IMK, Schloss Birlinghoven, D - 53754 Sankt Augustin, Germany stefan.conrad@imk.fraunhofer.de

Abstract. Any movie production needs a whole group of contributing authors and creative artists from various fields. The same should obviously be true for the making of a compelling VR scenario. Hence artists need to have direct ac-cess to the VR production process itself. In the VR domain however artistic and computational challenges are usually still severely interwoven. Thus nearly all of the (art) work is currently done by computer experts due to the unstructured workflow and the lack of creative tools for authors. In this paper we present two novel tools which we developed to adopt movie production procedures namely the VR Authoring Tool and the VR Tuner. The first is a tool to create a non-linear screenplay and storyboard analogue to the beginning of every movie production. The second tool, the VR Tuner, facilitates editing and post-production of VR scenarios for the first time. We conclude with a brief descrip-tion of our experiences during a first evaluation of the tools.

1 Introduction

Within the comparatively young field of highly immersive VR environments one can observe a steady improvement of display systems and interaction devices. On the software side sophisticated tools are available to create imagery and sound. But can this be enough to produce the best possible VR scenarios? Compared to the (Holly-wood) film industry this would mean that the production of a movie starts with build-ing up the scenery and filming right away and using the unaltered results as the final product. After more than a century of experience in movie production the film indus-try has developed a well-defined modular workflow. The procedure starts with the story outline, followed by the writing of the screenplay, casting and staging, filming and directing and finally the post-production with the editing of the film material. Each step is considered as an art form of its own with its specific refinements and own professionals. In current VR productions however certain specific artists are still more or less ex-cluded from most of the production process. It is the programmers who have to do the main work and hence have the strongest influence when it comes to the modeling of the story process and editing. The reasons for this are obvious, since the computers

Springer-Verlag

experts are usually the only ones who are able to put all the pieces for a working VR scenario together. Since they cannot rely on decades of experience in interactive story-telling, each VR installation is usually build up from scratch and ‘en bloc’. Authors of VR scenarios can only communicate their vision of scenery and story pro-gress in written form and hope that they are fully understood. A clearly defined plat-form does not exist. Once the VR scenario is completed and running for the first time it might be comparable to the ‘rough cut’ in movie productions. At this late point major changes in the VR production are usually impossible and even smaller im-provements (editing) are tedious and again can only be done by programmers. But the flexibility to make these changes and refinements will mark the difference between an acceptable and an exceptional production. An audience will less likely complain about shortcomings in technologies than about imperfect content. Hence it is time to enable authors and artists better of full access to this production process. We believe that this can only be done by the development of creative tools for these artists who for the most part not have programming skills. Our presented tools – the VR Authoring Tool and the VR Tuner facilitate important steps at the beginning and at the final stage of a VR scenario production, namely screenplay, storyboarding and post production with editing.

Fig. 1. The proposed production process of a VR scenario. The core production process is completed by a well-defined authoring and a post-production process. Every phase of this production process leads to a result which can be evaluated on its own

2 A Tool to support the VR Authoring Process

The goal of the presented VR Authoring Tool is to enable creative authors with no programming experience to create a non-linear script defining the whole story struc-ture including all possible forms of interaction and progression through the story without the help of programmers or technical experts. This script can be created and

Springer-Verlag

even tested with an integrated story player before the production of models, images and sound for the scenario has started. Beside the modeling of the story structure this tool also provides the functionality of a non-linear storyboard, where the author can communicate ideas about the presentation of the scenario to the VR production team.

Non-Linear Story Structure The concept of non-linear scenarios used for this VR Authoring Tool is based on parallel hierarchical graph structures. Graph structures are very intuitive and easy to grasp for authors which is demonstrated by the fact that they are typically used to illustrate and discuss non-linear processes [3, 4]. Furthermore they are widely used for authoring in different applications like an AI Editor [6], a Hypertext tool [2] and indi-rectly also in a storytelling tool [5]. The limitations of a single graph to model a com-plex non-linear scenario are obvious, because an author is forced to define all possible paths through the story in detail. But this argument does not hold when several graphs are introduced as parallel entities which are at first independent of each other but can be brought in context through an own scripting logic. While one central graph - the storygraph - roughly represents the overall progression of the story the other parallel graphs represent visible (e.g. scenery, doors, characters,…) as well as invisible (e.g. camera, timeline, apertures, plots…) objects. Every node of an object graph represents a certain state of that object. The overall state of the story is not defined by the actual state of the storygraph alone but by the actual states of all graphs. When we are talk-ing of states we think of story relevant states rather than of physics and geometry. In huge VR environments – in our case the CAVE or the i-CONE – a single state might enclose options for impressive spatial navigation. Nevertheless this could be a com-pletely irrelevant fact from a storytelling point of view. The connectors between the nodes imply possible transition inside a graph. Every node can be filled by the author with references to other objects as well as Event Scripts to define interdependencies between the referenced objects. An Event Script consists of a simple If-Then relation where a set of conditions (e.g. a state of a referenced object) must be fulfilled to exe-cute a set of results (e.g. a state of a referenced object or a transition to a adjacent node). Since there are basically no restrictions on how complex a story can be, the tool also offers the possibility to encapsulate entire parts of the story (e.g. self-contained story parts) within a - in that case - hierarchical higher node. Thus modeling of complex scenarios can be arranged more concisely by the author while the scenario structure itself gains stability and manageability for the associated story engine.

Non-Linear Storyboard Within each node, event specific information describing the scene or the object ap-pearance can be deposited by the author. This information – pure descriptive text and attached sample sounds, pictures or movies – serves as illustrative material for the modelers of the virtual world. The internal media player of the Authoring Tool en-ables the author to access the attached files for the most common media file formats.

Springer-Verlag

Testing with an Integrated Story Player A non-linear script can not just be read straight forward for evaluation as a linear movie script but entails different forms of progression and experience depending on the user’s interaction. Therefore an integrated Story Player is provided in this tool. This player enables the author to progress through the scenario and to simulate all the possible user interaction in order to test the behavior of the scenario. Cause the player provides the information about the story progress in a kind of debug mode, a scenario can be tested on the basis of the non-linear script alone, before any modeling or pro-gramming has started. For more information on the story player see [8].

Fig. 2. (left side) The VR Authoring Tool showing a storygraph and an object graph with an opened node. (right side) storyboard information of a node

The Authoring Tool in the further Workflow The benefit of the Authoring Tool is not limited to the authoring process in the begin-ning of a VR production. Assumed that the story engine of the target platform pro-vides an adequate interpreter, the non-linear script which consists of a very rigid xml structure can be used to control the progression of the scenario in real time. Conse-quently changes on the script which are made with the VR Authoring Tool will di-rectly effect the presentation of the scenario on the target platform. Hence last minute modifications and versioning is convenient and possible for non-programmers.

3 A Tool for Editing and Post-Production

According to the role of the VR Authoring Tool for story design and screenplay writ-ing, the VR Tuner provides functionality for fine tuning of an already produced VR story. In analogy to the post-production step in movie production the VR Tuner is used to apply changes to a story after the main development has already been con-cluded. The user is presented an interface combined of a 2D GUI on a wearable touchscreen computer and a set of hardware motor faders in a mixer console (see fig.3). The devices are brought into the immersive projection environment which

Springer-Verlag

makes them available in a running VR story. The touchscreen device can be detached from the mixer and carried around independently. For 3D interaction the touchscreen has a spatial tracker sensor attached. Being spatial aware, the touchscreens can be used like a scanner by pointing with it directly on 3D objects to retrieve their list properties. Selecting an object in 3D or in a 2D scenegraph and storygraph browser on the touchscreen computer, brings a list of the tunable properties of this object on the touchscreen. In addition, certain object parameters are assigned to motor faders. Using 2D controls on the screen and the motor faders, the user can now change the behavior and presentation of story objects. Tunable are for example light colors, material prop-erties and the pose of geometrical objects, volume and effects of sound objects, but also parameters of abstract objects like the behavior of the mover model, which is used in the story.

Fig. 3. The VR-Tuner hardware within an immersive projection environment

After the story representation has been changed according to the user needs all tuned parameters are made persistent in an XML based file format. Different sets of tunings can be loaded again at runtime to overwrite the original settings from the main devel-opment. The VR Tuner enables creative professionals who do not have to be VR or programming experts to tweak an already produced story. Small changes can be ap-plied in a WYSIWYG manner without the time consuming indirection via a program-mer or modeler. The interface has been designed for ease of use which has already been verified by professionals from stage productions who especially appreciated the use of the mixer as an intuitive, well known interaction method.

4 Conclusion

In this paper we argued for the necessities to modularize the process of VR produc-tions and to give non-programmers a stronger influence on this process to improve the quality of VR scenarios. We believe the best way to meet both necessities is the de-

Springer-Verlag

velopment of specific creative tools which flank the work of the VR programmers. We presented our two tools, the VR Authoring Tool for the creation of a non-linear screenplay and in advance testing of the VR scenario and the VR Tuner for the on site editing of a running scenario within a CAVE or i-CONE. First reactions of non-programming artists who tried to create non-linear screenplays for VR scenarios and computer games ranged from complete refusal to enthusiasm. It became apparent that support for authors is crucial when they engage in working with the tool for the first time due to the unfamiliar way of writing down stories. This initial blockage was eas-ily overcome by even the shortest workshop, after that authors used the Authoring Tool and the story player in a playful way. On the long run we are expecting them to come up with completely novel types of interactive scenarios. The tools themselves have proven to run soundly in a first small test scenario 'Taking care of the Little Prince's planet'. One of the major advantages for the production process is for example the chance to make working scenario progress alterations in less than a minute by simply modifying the storygraph with the help of the VR Authoring Tool – before that the modification of source code could easily last for hours. At present the tools are evaluated further under realistic commercial production conditions for the i-CONE VR game 'Entropy' which runs on the AVANGO VR platform [1, 7] and will be a permanent installation in the Technical Museum Vienna from May 2004 on.

Acknowledgements

This work was supported by the Ministry of Education and Research (BMBF) under grant No. 01IRA06A within the 'Virtual and Augmented Reality' project framework.

References

1. Beckhaus, S., Lechner, A., Mostafawy, S., Trogemann, G., Wages, R.: alVRed: Methods and Tools for Storytelling, International VR/AR Status Conference, February 19-20, 2004, Leipzig, Germany

2. Eastgate Systems, Inc.: Storyspace, http://www.eastgate.com/Storyspace.html 3. Ryan, M.-L.: Narrative as Virtual Reality, The Johns Hopkins University Press, Baltimore

(2001) 4. Samsel, J., Wimberley, D.: Writing for Interactive Media, Allworth Press, New York, (1998) 5. Silverman, B.G., Johns, M., Weaver, R., Mosley, J.: Authoring Edutainment Stories for

Online Players (AESOP): Introducing Gameplay into Interactive Dramas. ICVS 2003: 2nd International Conference of Virtual Storytelling, November 20-21, 2003, Toulouse, France

6. Stottler Henke Associates, Inc.: SimBionic, http://www.simbionic.com/overview.htm 7. Tramberend, H.: A Distributed Virtual Reality Framework, in Rosenblum, L., Astheimer, P.,

Teichmann, D. (eds.): Proceedings Virtual Reality 99, IEEE Computer Society Press, Con-ference, Houston, Texas (1999)

8. Wages, R., Grützmacher, B., Trogemann G.: A Formalism and a Tool for Diverging Re-quirements in VR Scenario Modeling, ICAT 2003: 13th International Conference on Artifi-cial Reality and Telexistence, December 3-5, 2003, Tokyo, Japan