3D Interaction

Mark Green

School of Creative Media

Introduction

• Investigate 3D interaction• how do we want to interact in 3D• look at different interaction techniques• try to develop a set of interaction primitives

that can be used to construct interaction techniques

• use SE to prototype interaction techniques

Interaction Tasks

• Earlier we looked at standard interaction tasks

• the main ones are: navigation selection manipulation combination

Interaction Tasks

• Are there other things that we would like to do in a virtual environment?

• In some sense the standard tasks are quite general so this is hard to answer

• if we look at some of the standard types of environments we might get a better idea

Viewing

• In simple viewing environments the main interaction technique is navigation

• the user moves about the data, but doesn’t interact with it

• not much here to add to our list

Indirect Manipulation

• User doesn’t directly interact with objects• instead sends commands to application,

which could change objects• use 3D versions of 2D IT to interact with

application, examples: menus, sliders, control panels, etc

• this is a start, gives us some ideas

2D Applications in 3D

• This leads us to 2D applications in 3D• does this make sense?• I’m not sure, I’ve change my mind on this

several times over the past 10 or so years• when we went from 1D text UI to 2D GUI

UI many of the applications transferred as well, helped spread 2D UI

2D Applications in 3D

• If we want to have a 3D desk top we need to support standard 2D applications

• need things like word processing, spread sheets, email, web, etc

• do these applications transfer to 3D? Does this make sense?

• How would we do this?

2D Applications in 3D

• There are two things to consider: output interaction

• many 2D applications are heavily text oriented, plus some simple graphics, tend to be page or screen oriented

• simple solution: texture map 2D screen onto a 3D polygon

2D Applications in 3D

• This is a quick and dirty solution, but somewhat works

• experimented with a 3D version of Netscape

• each page texture mapped onto a polygon, several linked pages visible at the same time

• 3D navigation used to move between pages

2D Applications in 3D

• It was interesting, but not very usable• need a better way of presenting the

information• on MS systems can extract data from many

applications using COM, could use this to get the data so it could be displayed in a different way, maintain compatibility with 2D version

2D Applications in 3D

• COM also allows us to modify data in office applications

• develop our own ITs and UI for these applications

• what would this type of UI look like?• Could we just transfer 2D ITs to 3D, a

quick and dirty translation

2D Applications in 3D

• We produced a set of 2D like ITs for 3D• based on the Motif widget library used in

X11• had many of the standard 2D techniques,

floated on a panel in 3D space• used in several applications, nice bridge

from 2D to 3D

2D Applications in 3D

• Do we want to do more than this?• How would we edit text in 3D?• Cut and paste may be possible, but selection

may be difficult, restrict motion to 2D surface of the document

• text entry still requires a keyboard, could speech recognition be an alternative?

2D Applications in 3D

• Provide a new 3D UI for these applications, be able to work with the data in 3D

• what would be the advantage? Integrated view of 2D and 3D data possibly easier to work with multiple

documents possibly better overviews of document structure

Direct 3D Interaction

• The next step is applications that support direct interaction with objects

• be able to select objects and perform operations on them

• what types of operations do we want to perform?

• To some extent this depends on the application

Direct 3D Interaction

• A general manipulation is geometrical transformation, ability to move the object, change size or orientation

• another operation is changing object properties, colour or application properties

• this could involve some type of property sheet or a 3D equivalent

Direct 3D Interaction

• Other operations involve editing the shape of objects, possibly producing new objects

• objects can be combined• in a vehicle simulator, the user needs to be

able to operate controls• grab a control and move it or maybe twist a

dial

Art and Entertainment

• Do these applications have their own types of interaction?

• This is an interesting question that is worth exploring

• in a sports game the user controls one of the players

• this is more suggestive than absolute in terms of control

Art and Entertainment

• The player has certain physical limitations and their own model of performance

• the user can suggest a direction of travel, but the player may not be able to move as fast as the user would like him to

• this is a co-operative form of interaction between the user and the application

Art and Entertainment

• What about the user controlling multiple characters?

• This is possible in some strategy programs, the characters follow the general directions of the user, but have some autonomy

• hard to have detailed control over more than one character

Art and Entertainment

• Are there things we would like to do in a purely artistic environment that wouldn’t be found in other types of environments?

• Most artistic environments have concentrated on visual appearance and exploration, there hasn’t been a lot of interaction

• I’m not sure why this is the case

Interaction Techniques

• Now we turn our attention to a detailed study of a particular interaction task

• selection, particularly command selection is an important interaction task that appears in most user interfaces

• the common way of doing this is with a menu, a well known set of interaction techniques

Menus

• Before menus, commands were entered by typing the name of the command and its parameters

• problem: need to remember the names of all of the commands

• menus provide a list of commands, don’t need to remember name, just know the right command to select

Footnote

• Menus remove part of the cognitive burden of using a program, don’t need to remember how to spell the commands

• menus provide a list of what there is, easy selection

• still need to know what each command does, temptation to keep adding commands, increases the complexity of the UI

Menus: 2D

• Start by looking at the important components of 2D menus

• many types of menus, vary along several dimensions

• menu type: static: always on screen in the same place dynamic: only appears when needed, can

change position (pull down vs. pop up menus)

Menus: 2D

• Standard menu shape is rectangular, with rectangular menu items

• can have circular menus, called pie menus, with each menu item having a wedge shape

• each menu item can have a text string or icon as a label

• this covers the visual elements of 2D menus

Menus: 2D

• The basic selection operation is a click within a menu item

• for pop-up and pull-down menus extra actions are required to display the menu

• only a few simple actions, either a click or a press-drag-release action

Menus: 3D

• How does this translate to 3D?• We need both a visual representation and

the actions required to select an item• the visual representation of menu items

transfers reasonably well: 3D text strings icons: simple 3D shapes

Menus: 3D

• Layout is more difficult• the menu must appear in 3D space where

the user can see it• this isn’t a problem in 2D, the menu is

always visible when displayed• in 3D, if the menu is placed behind the user

it won’t be visible

Menus: 3D

• 2D static menus have a fixed position on the screen

• in 3D there are two options for static menus: a fixed position within the environment,

possibly attached to an object a fixed position with respect to the user, travels

with the user

Menus: 3D

• In the case of pop-up menus there are two options: appear at the current cursor position, good for

selection appear a fixed distance in front of the user,

easier to find

• what is the visual appearance of a 3D menu?

Menus: 3D

• 2D menus now have sophisticated appearance, doesn’t always work well in 3D

• a fancy menu will hide all the objects behind it, take resources

• for ease of selection we want to menu close and relatively large, but this could hide a significant part of the environment

Menus: 3D

• For menus that float in space (not attached to an object) they should: have no background, the objects float in space if they have a background, it should be largely

transparent so users can see through it

• in this way the menu hides as little of the environment as possible

Menus: 3D

• Why do we care about this?• The user is concentrating on the application,

not the menu• if we hide the application data, the user

looses concentration, looses the context of the operation

• want to interfere as little as possible with the user’s task

Menus: 3D

• The shape of the menu depends upon how the items are selected

• want to use as little effort as possible• what are the selection actions?• This depends on the input device and the

interaction style, there are a range of possibilities

Menus: 3D

• The most direct approach imitates 2D selection, place a cursor in the menu item and press a button

• a 3D cursor is used for this, its position is controlled by one of the input devices

• this works well if the cursor is close to the menu, no need to navigate to the menu

Menus: 3D

• If a joystick or puck is used, cursor based selection works well

• if an absolute device, such as a Polhemus or Intersense is used, this may be more difficult, the menu must be within reach

• for these devices a pointing approach might be better

Menus: 3D

• In pointing a ray or line is controlled by the device: start of the ray at the device’s location direction of ray controlled by device’s

orientation

• by changing the hand orientation can sweep the ray through space

• the menu item hit by the ray is selected

Menus: 3D

• Ray based selected works for menus that can’t be reached, faster in this situation

• it can be difficult to aim at menus some distance away, ray can be replaced by a cone

• works well when rays are used to select objects, only one selection mechanism, less user confusion

Menus: 3D

• Both cursor and ray are physically based, use multiple degrees of freedom

• menu is just a list, one dimensional• just using one degree of freedom might be

easier• a good example of this is a ring menu, the

menu objects are distributed around a horizontal ring

Ring Menu

Menus: 3D

• Selection is based on rotating the ring• the object that is closest to the user is

selected• there is a selection zone at the front of the

menu, the item in this zone is selected if a button is pressed

• z rotation (or any axis) on a puck or joystick used to rotate the menu

Menus: 3D

• Ring menus work well as pop-ups• when a button is pressed the menu appears

at the current cursor position• a simple action can be used to move the

desired item to the selection zone• another button press selects the object• as easy as a 2D pop-up menu

Menus: 3D

• This last selection mechanism introduces the idea of a constraint

• we have a 6 DOF device, but we only use 1 DOF, we ignore the rest

• this makes it easier to perform the task, don’t need to rotate the menu about all three axis, just makes the task harder

Daisy Menu

Menus: 3D

• The last thing to deal with is feedback• before we press the button, need to know

we are getting the right item• with a ring menu this is part of the display,

the item in the selection zone• with the other selection techniques it’s not

near as obvious

Menus: 3D

• For cursor and pointing based techniques, highlight the item that would be selected if the button was pressed

• as the cursor moves into an item it is highlighted, when it moves out of the item the highlight is removed

• this is important since we can’t see the cursor when its inside an item

Menus: Summary

• We have looked at menus in detail, a technique from 2D UI

• we are now ready to dissect this technique, find the individual components

• find the pieces we need to build interaction techniques

• trace the actions required for a menu selection

Menus: The Components

• An event triggers the menu: for pop-ups: a button press that causes the

menu to appear for static menus: moving the cursor or pointer

into the menu region

• next a sequence of actions to select menu items

Menus: The Components

• for cursor need to know when inside an item

• for pointer, ray intersects the menu item• when this occurs we highlight the item, may

need to un-highlight a previous item• next we have the select action, which is

usually a button press

Menus: The Components

• We also need an escape route• what happens if the user doesn’t want to

select from the menu?• With a static menu just move outside of the

menu region• for a pop-up, do the select trigger without a

menu item selected

Menus: The Components

• The components: triggers cursor in menu item ray pointing at menu item highlight and un-highlight cursor or ray in menu menu display

Building Menus

• How do we know if the cursor is in a menu item?

• Need a general solution, can’t have a different technique for each menu

• this can be done by using bounding box techniques

• a bounding box is the smallest box that contains a graphical object

Bounding Boxes

• Most common form is axis aligned, the edges of the box are parallel to the coordinate axis

• find the smallest x, y and z coordinates, they form one corner of the box

• find the largest x, y and z coordinates, they form the other corner of the box

Bounding Boxes

• Easy to compute from object vertices• can also easily compute:

if a point is inside the bounding box if a ray intersects the bounding box

• these are the things we need• compute bounding box for each item, and

for the whole menu

Building Menus

• To trigger a static menu, check if cursor is in menu’s bounding box, or pointer intersects the menu’s bounding box

• use bounding box tests to determine the menu item that is selected

• draw bounding box as a way of highlighting a menu item, can use different colours

Interaction Techniques in SE

• Now that we have some idea of how ITs work, we will look at how we can construct them in SE

• The first thing we need is position and orientation data from input devices

• We need this to move cursors, make selections, move and orient objects, etc

Interaction in SE

• The attach and follow actions retrieve position data from an input device

• The format of these actions:attach device, x, y, z;follow device, x, y, z;

• The device parameter is the name of the device, this should be a joystick, tracker or some other device that produces 3D data

Interaction in SE

• The device name comes from the se.map file

• A device that produces 3D data is called a value device, its entry is:value internal_name device_name

• The internal_name is the name used inside our program or script

Interaction in SE

• The device_name is the name of the actual piece of hardware, for example:console: keyboard and mouse

joystick1: the first joystick

interd: the Intersense head tracker

interd2: the Intersense wand

• You may need to edit the se.map file to add a new device

Interaction in SE

• The last three parameters are variable names

• In the case of attach, these variables get the position of the device

• In the case of follow these variables get the relative position of the device, how far it moved on the last update

Interaction in SE

• We can use attach in the following way• we add the following line to se.map

value hand joystick1• Now we can do the following:

on tickattach hand, x, y, z;translate x, y, z;

end

Interaction in SE

• This small event handler can be used to move a cursor in response to the joystick

• This gives us a way of moving things, but not much interaction

• Need to be able to select objects as well• We know the position of the cursor, can use

bounding box test to see if its in an object

Interaction in SE

• The in action performs bounding box tests:in x, y, z, object;

• The first three parameters are the position of the cursor, or any other object

• The last parameter is a variable: If the cursor is inside an object, it will be the

name of the object Otherwise it will have no value

Interaction in SE

• Note that we can use select to determine if an object was selected, how can we use this?

• If we have a select trigger, can have

on select

in x, y, z, obj;

select obj;

trigger obj, “selected”;

end

Interaction in SE

• This event handler is placed in our cursor object, so x, y and z are the cursor position

• If the cursor is inside an object, the obj variable will contain the object’s name

• The select action will exit the event handler if nothing selected

• Trigger will send an event to the selected object

Interaction in SE

• So now we can tell an object its been selected

• There are two things left: Feedback How the object responds to selection

Interaction in SE

• Use a simple form of feedback based on bounding box

• Draw bounding box in wire frame when cursor is inside the object

• We will use white as bounding box colour• Could use different colours for different

types of feedback

Bounding Box Feedback

Interaction in SE

• Two actions are used for highlighting:highlight object, r, g, b;

unhighlight object;

• The first parameter is a variable, contains name of object to highlight or unhighlight

• r, g, and b give the colour of the highlight

Interaction in SE

• How do we use this?• When the cursor enters an object we want

to highlight the object• When the cursor leaves the object, we want

to turn off the highlighting• Problem: how do we know when the cursor

leaves the object?

Interaction in SE

• in tells us when we are inside of an object, but it doesn’t tell us if we have left an object

• Need to remember last object we were in, unhighlight this object

• Then test to see if we are inside an object, and highlight that object

Interaction in SE

on tick

attach hand, x, y, z;

translate x, y, z;

unhighlight obj;

in x, y, z, obj;

select obj;

highlight obj, 1, 1, 1;

end

Interaction in SE

• If the cursor is inside an object, we first unhighlight the object, then highlight it again

• But when we move out of the object, we will unhighlight it, but it won’t be highlighted again

• This does what we want

Interaction in SE

• Now that we can select an object, what do we do with it?

• There are two possibilities: The user has selected a command, this

command applies a one shot change to the object

This is the start of a continuous manipulation, such as changing the position or orientation of the object

Interaction in SE

• In the first case, the cursor object sends an event to the selected object, which then changes in some way

• The cursor object needs to know the event to send, based on a state variable

• Cursor object received an event (from a menu?) that sets this state variable

Interaction in SE

• In the second case, the selected object will respond to the input device

• On each tick it reads the device and performs some action until it’s told to quit

• The action to perform is again determined by a state variable, in the selected object this time

Interaction in SE

• We need a way of examining state variables and then determining what to do

• This can be done with the triggerif action, a combination of the if and trigger actions:triggerif conditional expression, object, event;

• The first part looks like an if action, there is an expression that evaluates to zero or non-zero, will usually involve a comparison operator

Interaction in SE

• The second part contains the event to be sent if the condition is true

• The object is a variable that contains the name of the object the event is sent to

• The variable me contains the name of the current object

• Now lets see how this works

Interaction in SE

• Produce a simple environment, an object that can be selected, then translated or rotated

• use the following button as triggers: one: translate the selected object two: rotate the selected object three: stop translating or rotating

Interaction in SE

• The object has a variable, state, determines which action it’s performing

• on each tick examine state, then transfer to appropriate event handler for current action

• we can use triggerif for this, testing the value of state

Interaction in SE on tick

triggerif state == 1, me, “move”;triggerif state == 2, me, “turn”;

endon move

follow hand, x, y, z;move x, y, z;

endon turn

dorient hand, rx, ry, rz;spinx rx;spiny ry;spinz rz;

end

Interaction in SE

on select1set state, 1;

endon select2

set state, 2;endon stop

set state, 0;endon first

set state, 0;end

Interaction in SE

• The cursor object sends the select1, select2 and stop event depending upon the button the user presses

• for the first two buttons, check to see if the cursor is inside an object before the event is sent

• for the last event, need to remember the object performing the interaction

Interaction in SE on select

in x, y, z, obj;select obj;set last, obj;trigger obj, “select1”;

endon twist

in x, y, z, obj;select obj;set last, obj;trigger obj, “select2”;

endon three

trigger last, “stop”;end

Interaction in SE

• The tick handler for the cursor retrieves the cursor position, stores in variables x, y and z

• it also performs the highlighting and unhighlighting

• these event handlers only need to respond to button presses

Menus in SE

• We can also do standard menus in SE• To see how this works we will do a simple

pop-up menu that has three menu items• The user presses button two to pop up the

menu• The cursor is moved into an item, button 1

is pressed, the menu then disappears

Pop-up Menu

Text

• The first thing that we need for this example is some text

• We could use a modeler to produce the text, export it as a text file, …

• This is a lot of work for just a text string, so we have a text command that is used to produce 3D text

Text

• The text command starts with the word text, followed by the name of the text object, ends with a line containing end

• Number of options that define the text object

• The main option is text followed by the text string to be displayed, in quotes (“)

Text

• Other options include:position x y z

direction dx dy dx

up dx dy dz

size s

• The position option gives the location of the text string, can be transformed in an SE object

Text

• The direction of the characters in the string is given by the direction option

• The up option gives the direction of the up vector for the string (the vertical direction for the characters)

• The size option is used to scale the text string

Menustext text1

text "item one"

position 0.5 0.2 0.8

direction 0.2 1 0

size 0.3

end

object menu1

add text text1

on select

“do something”

end

end

Menus

• The add text option inside of an object command is used to add a text string to it

• All three items are basically the same. Only the z coordinate of the text position changes

• Now we need to arrange some way of popping the menu up and down

• We use a scene for this

Menusscene first

on entryactivate “cursor”;activate “xyz”

end

on twistactivate “menu1”;activate “menu2”;activate “menu3”;

endon select

deactivate “menu1”;deactivate “menu2”;deactivate “menu3”;

endend

Scenes

• When a script file has scenes none of the objects are active when the environment starts

• If an object isn’t active it isn’t displayed and it doesn’t respond to events

• The entry event handler makes the cursor and light active using the activate action

Menus

• When the user presses button 2, the twist event is generated

• The event handler for twist activates the three menu items

• When this happens the menu items become visible and they can be selected

Menus

• Each object has an event handler for the select event

• This event handler processes the command, more on this later

• The select event handler in the scene deactivates all three menu items, so they are no longer visible on the screen

Menus

• The scene object handles the popping up and down of the menu

• Done in response to button presses• Each menu item is responsible for its

command, whatever that is• But how is this done? All three menu items

get the select event

Menus

• Menu items need to know if they have been selected

• In gives us the object selected, but no way of telling if it’s the current object

• We use the inObject action to determine if the menu item has been selected

• This is a simple version of in

Menus

• The format of the inObject action is:inObject expression, x, y, z, variable;

• The first is the name of the object, either a text string or a variable

• If the point (x,y,z) is inside this object, the second variable is set to one, otherwise it is set to zero

Menus

object menu1add text text1on select

attach hand, x, y, z;inObject me, x, y, z, flag;select flag;trigger “menu1”, “selected”;

endon selected

print "Item one selected”;end

end

Menus

• The select event handler gets the cursor position

• Uses inObject with the me variable to test if the cursor is in this menu item

• The select action tests the return from inObject

• If selected the selected event is sent