Super-Resolution / Image Registration Overviekasim/eng_design/vr/Lecture19-User-interfaces.pdf ·...
Transcript of Super-Resolution / Image Registration Overviekasim/eng_design/vr/Lecture19-User-interfaces.pdf ·...
Department of Signal Processing
User interfaces
SGN-5406 Virtual Reality 2012Atanas Boev
based on material byStanislav Stankovic and Ismo Rakkolainen
1
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Outline
SGN-5406 Virtual Reality 2012
The UIproblem History Modern UI
design UI in VRThe users Humanfactors
Mechanical UI
Computer UI
GUI elements
What is UI? User types
Subjectivefactors
Subjectiveexperiments
Evaluation
Cyber-sicknessProblems
Touch UI
Navigation
Manipulation
2
Modern UItrends
Modern pitfalls
Department of Signal Processing
OVERVIEWUser interface
3
SGN-5406 Virtual Reality 2012
Department of Signal Processing
What is UI?
• UI - the “way” users interact with the system• Input - users change the state of a system• Output – system indicates its state to the users
• The “way” (UI paradigm) – includes:• UI protocol – how the control behaves• UI presentation – what hardware controls are used
• Also called HMI – human-machine interface• UI layer – the software that implements the UI behaviour• UI layout - how controls are positioned• etc...
SGN-5406 Virtual Reality 2012
4
Department of Signal Processing
Problems with UI design
• All UI design isdictated by technology• No technology is perfectly suited
for human anatomy
• There is no “natural” way ofinteraction• All interaction are learned habits• Pen and paper is natural to
people because they are trainedto use it.
SGN-5406 Virtual Reality 2012
5
Department of Signal Processing
Problems with UI design
• People do not know whatthey want• People know what they do not
want
• Focus groups fail• Most people like voice
recognition, but will not use it.• Most people like DVB-H, but
will not use it.• Most people like AR maps, but
will not use it.
SGN-5406 Virtual Reality 2012
6
Department of Signal Processing
Problems with UI design
• UI design should be done by designers• However, designers are not sure what what people want• Still, designers know (slightly) more than engineers
• UI quality is hard to measure• Quality is hard to measure• Subjective quality is hard to quantify• Performance can be quantified• Subjective tests are not an easy thing to do• Subjective tests are not an easy thing to analyse
SGN-5406 Virtual Reality 2012
7
Department of Signal Processing
MECHANICALUI history
8
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Time
• Round dial• Imitates sun movement• “Noon” is on top
• 12 sections and threehands• Hour hand• Minute hand (appeared in 1500)• Sweep hand (appeared in 1800)
• Multiple dials can bepresent
SGN-5406 Virtual Reality 2012
9
Department of Signal Processing
Unconventional dials
SGN-5406 Virtual Reality 2012
24h dial Decimal dial Chemical elements dial
10
Department of Signal Processing
Digits on mechanical clocks
SGN-5406 Virtual Reality 2012
11
Department of Signal Processing
The corpus clock
• Mostly mechanical• Grasshopper escapement
• Time is displayed viabacklit slits• Slits open and close• Backlight passes trough• Position of light indicates
the time
SGN-5406 Virtual Reality 2012
http://youtu.be/cCqGtvTA36k
12
Department of Signal Processing
Steering
• Steering wheel• First used in 1894• Spoke design allows seeing
the dashboard• Integrated controls
• Car pedals• Brake in the centre• Clutch away from the
throttle minimizes errors• Foot rest
• Ergonomic design• Not “untuitive”
• Follows car principes, ratherthan mind intuition
SGN-5406 Virtual Reality 2012
Clutch, brake, throttle
13
Department of Signal Processing
Steering wheel
• Early cars were steeredwith tiller• Like in a boat
• Modern versions exist• Steering wheel is the
standard• Quite effective• Drivers are trained to use it
• “Meet in the middle”• Technology adapts to man• Man adapts to technology
SGN-5406 Virtual Reality 2012
Clutch, brake, throttle
14
Department of Signal Processing
Formula one wheel
SGN-5406 Virtual Reality 2012
• Smaller wheel, many controls• Meant for high speed and low manoeuvrability
15
Department of Signal Processing
The “design for interface” problem(a.k.a. Asimov’s pitfall)
• The reason for building humanoid robots?• Instead of making robot car and robot tractor, make
normal car and normal tractor and a robot to drivethem (Asimov’s books)
• Technology advances much faster thaninterfaces• By the time humanoid robot is possible, self driving
car will be very cheap to do
• Interface should adapt to technology• Technology should NOT adapt to the interface• Example – handwriting recognition is a flop, voice
recognition… maybe?
SGN-5406 Virtual Reality 2012
16
Department of Signal Processing
EARLY COMPUTER UIUI history
17
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Early computer UI
SGN-5406 Virtual Reality 2012
Input Output
Punched card – a column of holesrepresents an entry: command,number or symbol
Teletype – receives numbers,prints characters, basically acomputer-controlled typewriter
18
Department of Signal Processing
Early computer UI
• Evolutionary approach• Early computers were
seen as typewriters withTV screen
• Meant to replacecalculators
• Meant to replacetypewrites (text editing)
• Interface devices• Keyboard• Display• Joystick, mouse
SGN-5406 Virtual Reality 2012
19
Department of Signal Processing
Text based UI
SGN-5406 Virtual Reality 2012
20
• Keyboard shortcuts – each letter represents a command (saveskeystrokes)
• Same letter means different commands depending on the context
Department of Signal Processing
Text based UI
• All input is done trough keyboad• Might work in
• Popular with text-centric tasks• Text editing – vi, vim, e, jed...• Line based text editors – work on one line at a time• Screen based text editors – freely go up and down
trough the text
• Fast interaction (faster than mouse)• Steep learning curve
• When programming was highly specialised task• and “teletype operator” was a job position
SGN-5406 Virtual Reality 2012
21
Department of Signal Processing
COMPUTER GUIUI history
22
SGN-5406 Virtual Reality 2012
Department of Signal Processing
GUI elements - Windows
• “WIMP” – windows, icons, menus, pointer• The classical mouse-driven GUI• Developed by Xerox PARC• “Popularised” by Apple
• GUI elements• Windows• Menus• Icons• Controls• Tabs• Toolbox
SGN-5406 Virtual Reality 2012
23
Department of Signal Processing
GUI Windows
• Single window mode
• Tiling windows• Windows do not overlap
• Stacking windows• Windows can overlap• Windows can scroll their
contents• Program does not need (but
is good to) to adapt todifferent window sizes
SGN-5406 Virtual Reality 2012
Single window mode
Tiling windows
Stacking windows
Department of Signal Processing
Icons
• Icon – a graphicalrepresentation of a file• Files appear as “objects”• Executable files – custom
icons• Data files – one icon per
file type
SGN-5406 Virtual Reality 2012
Win 3.1 icons, 32x32px
OSX icons, 1024x1024 (HiDPI) icons
Win XP icons, 64x64px
25
Department of Signal Processing
Toolbox
• Floating menu• Meant for graphic editors
(Photoshop, Corel, etc)
• “Tool options”• Additional context menu• Options of the currently
selected tool
SGN-5406 Virtual Reality 2012
Toolbox
Tool options
26
Department of Signal Processing
GUI widgets
• Widgets• The “active elements” of a GUI
• Also, their appearance and behaviour• “widget” = “window gadget”
• List of GUI widgets• Button• Window controls (maximize, minimize,
close)• Dropdown menu• Text box (plain text, password)• Date selection• Etc, etc, etc…
SGN-5406 Virtual Reality 2012
27
WebOS widgets Symbian widgets
Windows 95 widgetsOSX “Aqua” widgets
Department of Signal Processing
Case study – Microsoft Word
• Word 5.0 – screen-based text interface• UI elements
• Document window• Command menu• Status bar
SGN-5406 Virtual Reality 2012
28
Department of Signal Processing
Case study – Microsoft Word
• Word 5.5 –text+mouse
• UI elements• Drop-down
menu• Document
window• Command menu• Scroll bars• Status bar• Context help line
(botton)• GUI in text mode
• Dropdown is aGUI element
• Scrollbars areGUI widgets
SGN-5406 Virtual Reality 2012
29
Department of Signal Processing
Case study – Microsoft Word
• Word 6.0, Word 95,Word 2000• GUI interface• No major UI changes• Mostly re-skinned
version
• MS had dominantposition on themarket• No need for innovations
SGN-5406 Virtual Reality 2012
Word 6.0 / Win 3.1
Word 95 / Win 95
Word 2000 / Win 2000
30
Department of Signal Processing
Case study – Microsoft Word
• Word 2007• Ribbon interface• Similar to tabbed
combo-box• Replaces drop-
down menus• Ribbon adapts
(“reflows”) tovarious displaywidths
• Easier to findcommands(compared todropdown)
• Ribbon can be“folded” bydouble-clicking ona tab
SGN-5406 Virtual Reality 2012
31
Department of Signal Processing
TOUCH-BASED UIUI history
32
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Touch-based UI
• Allows interaction with nokeyboard• “Keyboard space can be used as screen
estate”• Also, cheaper and more durable• Does not have tactile feedback
• Allows direct manipulation• Requires bigger touch-targets (finger is
less precise than cursor)• Fingers obstruct the drawing – needs
special text-selection tools
SGN-5406 Virtual Reality 2012
33
Department of Signal Processing
Keyboard alternatives
• Keyboard overlays• External keyboards• Touch screen-based tactile overlay
• Gesture-based keyboards• Swype – allows drawing trough letters
insteac of tapping them• Graffiti – special alphabet that is easy to
recognize by the system• Simplified version of handwriting
recognition• Requires training
SGN-5406 Virtual Reality 2012
34
Department of Signal Processing
Home screens
SGN-5406 Virtual Reality 2012
35
iOS home screen• Status bar / notifications• Icons• Dock
Android home screen• Status bar / notifications• Icons / widgets• App drawer
Windows phone home screen• Live tiles• Vertically scrollable• Comic book style frames
Department of Signal Processing
SUBJECTIVE FACTORSThe users
36
SGN-5406 Virtual Reality 2012
Department of Signal Processing
User types
SGN-5406 Virtual Reality 2012
37
Department of Signal Processing
Age
• Users of different age groups have differentneeds• Different abilities• Different style
• Some factors are obvious• bigger buttons for old people• brighter images for kids
• Some factors are not obvious• Older people use index finger to push buttons• Younger people use thumbs• Older people prefer to pay cash• Younger people do not wear watches
SGN-5406 Virtual Reality 2012
38
Department of Signal Processing
Social background
• In some cultures public photography notacceptable• “Female college students carrying camera phones
on campus in any college in Saudi Arabia couldface a SR500 fine and 3-year suspension”
• Camera sound cannot be turned off in Japan
• In some countries insulting the governmentis a crime, downloading mp3 is not
• In some other countries, it’s the opposite
SGN-5406 Virtual Reality 2012
39
Department of Signal Processing
User experience
• User experience (UX) - the way a person feelsabout using a product, system or service• Quality of experience (QoE) – subjective measure a
system performance (“Do users approve?”)• Quality of service (QoS) – objective measure of system
performance (“Do engineers approve?”)
• User-centered design (UCD) – UI design whichis focused on user need and limitations• Design is a function of the user expectations and needs• It is critically important to perform correct test and
analysis• Otherwise design will be based on what users seem
to want• … if the users know what they want in the first place.
SGN-5406 Virtual Reality 2012
40
Department of Signal Processing
Subjective tests
• Subjective tests have to adhere to a well-documented experimental protocol e.g.• ITU-T BT.500 Methodology for the Subjective
Assessment of the Quality of Television Pictures• ITU-T BT.1438 Subjective video quality assessment
methods for multimedia applications
• Experiments need to follow a structuredsequence for all participants• A trial is a single instance of the experiment.• A sequence of repeated trials constitutes a session.• Sessions (and sometimes trials) are separated by rest
periods for the participant in the study
SGN-5406 Virtual Reality 2012
41
Department of Signal Processing
MODERN UI TRENDSModern UI design
42
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Tiling window managers
• Single window interfaces• Most mobile OSes show one
app on full screen
• Tiling windows• Windows 8 modern UI allows
running multiple apps side-by-side
• Cascading windows• Galaxy note supports
“floating widgets”• Similar to MacOS 6 “Desk
accessories”
SGN-5406 Virtual Reality 2012
43
MacOS Deskaccessory
Floating notepad window
Win8 UI, two apps side-by-side
Department of Signal Processing
“Responsive” design
• UI elements actively resize• If the element is selected• If the windows size changes• Used in web-design
• The same webpage works onmany screen resolutions
• Used in mobile OS• The same software works on
phones and tablets
• Example – the “accordion”• Widgets stretches on selection and
contracts on losing focus
SGN-5406 Virtual Reality 2012
44
Accordion widget, “Header 2” is selected
Department of Signal Processing
Infinite canvas
• Invented by Scott McCloud• Modern technology allows for having a
multidimensional narrative
• Possibly used as a basis of Metro Modern UI
SGN-5406 Virtual Reality 2012
45
Department of Signal Processing
Drag-to-refresh
• Refresh a list of events by dragging it downwards• First used in Tweetie (bought by Twitter)• Nowadays used by many mobile apps
SGN-5406 Virtual Reality 2012
46
Department of Signal Processing
Skeuomorphism
• Skeuomorphic design• Create the outlook of one
(expensive) material usinganother (cheaper) material
• In GUI – using ultra-realistic widgets,mimicking real objects• Shadows• Textures• Mimicking realistic object
behaviour (e.g. rotating knob)
SGN-5406 Virtual Reality 2012
47
Department of Signal Processing
MODERN UI PITFALLSModern UI design
48
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Skeuomorphism
• Skeuomorphism can easily be overdone• Instead of “luxurious” created “cheap” feel• E.g. fake “leopard” prints
SGN-5406 Virtual Reality 2012
49
UnderstatedOverstated
Department of Signal Processing
Volume control
• iOS “mute” button does not mute all sounds• Android has 6(!) separate volume controls
• Media• Notifications• Alarms• System (keyclick)• Headphones• Calls
• One does not simply mute (into Mordor)• Lots of embarrassing situations• Nokia “profiles” disappeared in WP7+
SGN-5406 Virtual Reality 2012
50
Department of Signal Processing
Acceleration sensors
• Accelerations sensors are often overused• Games are controlled with device tilt• Hard to do precision control• Hard to follow the action on the screen• Tilt-control lunar lander (?)
• Display orientation• Two modes: locked/unlocked• No way to set display orientation• Hard to use while laying on the sofa• Impossible to use in zero-g
SGN-5406 Virtual Reality 2012
51
Department of Signal Processing
VR EXPERIENCEUser interfaces in virtual reality
52
SGN-5406 Virtual Reality 2012
Department of Signal Processing
VR experience design
SGN-5406 Virtual Reality 2012
53
Factors which affect the VR experience
• Aims of VR experience design• Immersive experience• Easy to perceive (no sensory overload)• Easy to control (easy to grasp and navigate)
Department of Signal Processing
Interaction tasks
• Traditional interaction• Imitation of real objects (see skeuomorphic design)• Imitation of familiar 2D GUI (see 3D dropdown menus)
• Traditional interaction might not be useful in a VE• E.g. using walking for map or areal navigation – might be too tiring• Using text based menus in VE – text might be seen from hard-to-
read angles• VE navigation needs new concepts and paradigms
• VE tasks• Navigation: travel + wayfinding• Selection: pointing, choosing one or more objects• Manipulation: Grabbing, object position, orientation, scale• Communication: with users, with UI agents• Data input: symbols, text, numbers• Commands: menus, changing of system state• Visual information: system status
SGN-5406 Virtual Reality 2012
54
UI agent
Department of Signal Processing
NAVIGATION IN VRUser interfaces in virtual reality
55
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Navigation
• Navigation = travel + wayfinding• Wayfinding – finding the path to go from A to B• Travel – actual moving of the point-of-view
• Travel methafors• Natural methaphors
• walking, bicycling,• Also Disney raft ride
• Steering – specifying the direction• Steering wheel, joystick, gaze directed
• Target based – e.g. teleport• Point at place, enter coordinates, choose from list
• Orbital movement
SGN-5406 Virtual Reality 2012
56
Department of Signal Processing
Navigation
• Wayfinding metaphors• Natural metaphors
• Landmarks• Previous knowledge• Memorable place names
• 3D map-based• Bird-eye views• Constrained travel – cannot leave the predefined path• Path-following – leave trail, place markers
SGN-5406 Virtual Reality 2012
57
Department of Signal Processing
VR Navigation UI
SGN-5406 Virtual Reality 2012
58
Point places on aminiature model of theworld
Department of Signal Processing
VR Navigation UI
SGN-5406 Virtual Reality 2012
59
Select steering direction
Department of Signal Processing
MANIPULATION IN VRUser interfaces in virtual reality
60
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Selection / manipulation
• Selection of direction• Using pointer• Using gaze• Using crosshair• Torso-directed (slant of the body towards desired direction)• Landmark-directed
• Selection of item• Point to select - direct contact between object and avatar• Using 3D cursor• Using aperture – e.g. two fingers• Using menu• Using name – e.g. speech recognition
• Manipulation• Apply force to virtual objects• Gesture-based control of size (e.g. pinch to zoom), color, texture• Using menu – modifying object properties
SGN-5406 Virtual Reality 2012
61
Department of Signal Processing
EVALUATIONHuman factors in VR
62
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Human factors in VR
• Multimodal interfaces (with graphics, sound and hapticfeedback), realistic modeling, and increased systemperformance can produce better-quality simulations
• Are VR systems, UIs really more efficient for a specifictask?• Claims about increased performance need to be backed up by
proper user studies• User’s performance when interacting with the simulation• User’s response to the technology in order to iteratively improve the
VR system or the particular application design• It is necessary to understand why some user responses lead to
simulator sickness, what are its causes, and what can be done tominimize its effects
• Courses at TUT: http://www.cs.tut.fi/ihte/• Design and evaluation of 3D-UI: http://www.3dui.org/
SGN-5406 Virtual Reality 2012
63
Department of Signal Processing
Human factors research
• Human factors research consists of systematic studiesby multi-disciplinary teams of engineers andpsychologists to gauge• Which tasks are more suitable for users• Which user characteristics influence the VR simulation performance• How VR technology should be improved to better meet user needs• What kind of designs enhance user performance• The negative societal impact from the users’ misuse on the
technology, etc.• No comprehensive model of human behavior exists,
owing to its multidimensionality as well as largeindividual variability
• The fairness of a simulation is qualitative and it cannotbe easily quantified mathematically.
• Also, determining the performance of a VR simulation issomewhat subjective
SGN-5406 Virtual Reality 2012
64
Department of Signal Processing
Human factors research
• Human factors research can be divided into• Usability – studies how to improve a system or
application• User performance – aims to measure the user’s
response to a given simulation and a particularhardware system
• User safety – aims to better understand simulationsickness causes and effects as well as increase theuser’s safety
• Sociological studies – aim to measure factors relatedto effects of VR on other users and the society
SGN-5406 Virtual Reality 2012
65
Department of Signal Processing
Selection / manipulation techniques
SGN-5406 Virtual Reality 2012
66
Department of Signal Processing
Menu selection in VR
SGN-5406 Virtual Reality 2012
Department of Signal Processing
CYBERSICKNESSHuman factors in VR
68
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Effects of VR Simulations onusers
• Indirect effects• neurological, psychological, sociological, or cybersickness• E.g., addiction, lack of real human relationships, narrow social life, etc.
• Direct effects• affect mainly the user’s visual system (dominant),• also the user’s auditory, skin, and musculoskeletal systems
• Examples• Visual system can direct high-intensity light at the user’s eyes. The exposure
will result in corneal burns, retinal burns, and other injuries• The laser used in miniature wearable retinal displays• Bright lights pulsed at low frequency may trigger an induced “absence” state
or seisure• Auditory system affects the user if the simulation noise level is too high• Haptic interfaces can apply high level of forces or push the user’s limbs
beyond anatomical range limits• Motion platforms that move ankles beyond normal rotation angles• Increased risk of skin disease when VR interfaces (e.g., HMDs, gloves) are
shared between several users
SGN-5406 Virtual Reality 2012
69
Department of Signal Processing
Cybersickness – what is it?
• Cybersickness• perhaps the most troublesome effect of VR simulation to the user• A form of motion sickness that results from VR interaction
• Main symptoms• eye strain• disorientation• postural instability• sweating• drowsiness• Nausea
• Cause• information from several sensorial channels is not in agreement
• Cybersickness vs motion sickness• Motion sickness may be induced without immersion (e.g. by riding a
roller-coaster in an amusement park• Cybersickness is caused by exposure to virtual environments (e.g.
by immersive visual environment)
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Cybersickness - causes
• Main reason – inner ear• Conflicting multimodal information which involves the vestibular
sensors• Sense of balance perceives one thing, eyes/ears/skin perceive
another• Causes
• Rendering errors• System latency• Misaligned HMD optics• Vection
• Fast scene movement on a large FOV displays• Movement occupies central and peripheral vision• Eyes perceive movement, vestibular sense does not
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Cybersickness - susceptibility
• Age-dependent• People aged 2-12 are the most susceptible to
cybersickness• People at age 25 are half as susceptible than people
at age 18
• Gender dependent• Female users 3X more susceptible than male users
• Medication dependent• Mind affecting medications affect susceptibility to
cybersickness• Small doses of alcohol may decrease it• Large doses of alcohol may cause motion sickness
SGN-5406 Virtual Reality 2012
Department of Signal Processing
Cybersickness - adaptation
• Brain can adapt to contradicting sensorialstimuli• Repeated VR exposures can cause adaptation, and
decrease susceptibility to sybersickness• Brain learns to treat abnormal stimuli as normal
• Speed of adaptation can be increased• By active interaction• Gradual and incremental VR exposure
SGN-5406 Virtual Reality 2012