eNTERFACE 05-06-07-08 … A “historical” perspective T. Dutoit – Polytech’Mons
Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory...
Transcript of Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory...
![Page 1: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/1.jpg)
Experimental User Interfaces
eNTERFACE'13
Tapio Takala
Aalto University Dept. of Media Technology
http://www.cs.hut.fi/~tta/
![Page 2: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/2.jpg)
eNTERFACE 2013 – Takala
Contents o Why experimental interfaces?
o Towards common methodology
o Cases from research experiments
o Trends for future
2
![Page 3: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/3.jpg)
eNTERFACE 2013 – Takala
1. Why experimental UI ? o dominant design now
o gestural 2D interfaces: mouse / touch
o sensory limitations in current UI, new devices
o missing established methodology
3
![Page 4: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/4.jpg)
eNTERFACE 2013 – Takala
Current dominant paradigm o Basic features
o 2D visual display + point & click device (mouse) o direct manipulation o hierarchical window structure o virtual devices (widgets)
o menu o button o text box o scroll bar
o event-based control loop o window manager o device handlers
o gestural interaction techniques o drag & drop o selection by ”sweeping” o opening by double click
WIMP = Windows - Icons - Menus - Pointers
4
![Page 5: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/5.jpg)
eNTERFACE 2013 – Takala
Isn't this enough ?
5
o "A Personal Computer for Children of All Ages" (Alan Kay's Dynabook vision, 1968)
using iPad today, after 40 years of development
![Page 6: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/6.jpg)
eNTERFACE 2013 – Takala 6
not just a notebook, but o multisensory / multimodal
o different concurrent devices / kinds of use o image, sound, video… o graspable / movable objects (tangible UI) o wearable computing o smart environment watching the user
o continuous (non-discrete) control o no command dialogue, but continuous flow o computer may act as an initiative agent
o tied to the real environment, "physical computing"
Natural interaction is…
![Page 7: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/7.jpg)
eNTERFACE 2013 – Takala
Use of senses in current interfaces
o very asymmetric: tactile input + visual output; sound underutilised
o exception: multimedia content (sound and images in and out)
o NOTE: the fifth ”feel” sense actually covers several different senses
input (to computer) output (display) sight webcam
offline images UI (widgets) and content (text, images, video, etc.)
hearing speech recognition content recording
alert signals (UI) soundtracks (content)
smell ? ?
taste ? ?
touch keyboard mouse / finger
force feedback (in some game consoles)
7
![Page 8: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/8.jpg)
eNTERFACE 2013 – Takala
New input devices o current standard devices: keyboard + mouse
o recently being replaced by touch pad
o available but less used o pen/stylus, joystick, track ball o 3D trackers o force/torque handles o speech recognition
o new potential o image capture o non-speech sound o motion sensors (accelerometer) o positioning (GPS) o biosensors (EMG) o chemical sensors, smell/taste o etc.
8
![Page 9: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/9.jpg)
eNTERFACE 2013 – Takala
'Put That There' (Bolt 1980)
o pointing gestures
o speech commands
o large screen display
o multi-user interface
Why these still aren't standard features of a UI ?
9
![Page 10: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/10.jpg)
eNTERFACE 2013 – Takala
2. Towards methodology in software o Computer graphics standardization
o Model-View-Controller framework
o Device abstractions
o A mental experiment
10
![Page 11: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/11.jpg)
eNTERFACE 2013 – Takala
CG standardization
o drawing in device independent coordinates
o input part not well established
drawingprimitives
displaydriver
applic.prog.
model
logicalinput
devices
inputdriver
segment store(display list)
APPLICATIONDEVICE
INDEPENDENTGRAPHICSLIBRARY
OS / WINDOW MANAGER
echo
pick
11
official: GKS (1985) de facto: OpenGL (1992)
![Page 12: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/12.jpg)
eNTERFACE 2013 – Takala
The MVC conceptualization o often used in web applications
o works well with common widgets o also with hierarchical structure
o may become difficult in more general use o the controller may become
very complex
12
![Page 13: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/13.jpg)
eNTERFACE 2013 – Takala
Input device abstractions o a useful framework for conceptualization and generalization
o what information is transferred from human to machine ? o state or its change (on/off) è boolean o one from a set of alternatives è integer [enumeration] o text è character [string] o continuous numeric value è float o location (2D/3D) è vector [x,y] o reference to a pointed object è id (name/address/number)
o may be implemented in many ways by different devices o switch, function key, alphanumeric key, slider, cursor on screen
(indicator of mouse), drawing tablet, 3D tracker, camera, speech recognizer …
o Try it out: implement devices by each other J
13
![Page 14: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/14.jpg)
Exercise: fill the empty slots
native ? ? ? ?
? native ? ? ?
? ? native ? ?
? ? ? native ?
? ? ? ? ?
text string (char)
real number���(float)
choice (one out of���few alternatives)
2D position���(x,y)
information for!the computer:!
object picked���from screen���(identifier/name)
![Page 15: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/15.jpg)
eNTERFACE 2013 – Takala
Conclusion (for today) o Real life communication too complex
to be standardized o attempts in early 80's to do the same for interaction as for
graphics, with limited success o works for limited applications and modalities, e.g. the
desktop/notebook metaphor
o However, abstractions help to keep it simpler o software modularity: separate data manipulation from action
control, if possible
o Be ready for very different paradigms in the future
15
![Page 16: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/16.jpg)
Paradigm changes in UI software
Current
o single-threaded I/O
o discrete tokens
o precise tokens
o sequence, not time
o explicit user commands
Future
o parallel, asynchronous dialogues; may be interrelated
o continuous inputs and responses (plus discrete)
o probabilistic input, not easily tokenized
o real-time requirements, deadline-based
o passive monitoring of the user
source: Butler, Jacob & John CHI’98
16
![Page 17: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/17.jpg)
eNTERFACE 2013 – Takala
3. Sample cases o Musical applications
o New art forms
o Motion based games
o Eyes-free interfaces
17
![Page 18: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/18.jpg)
eNTERFACE 2013 – Takala 18
DIVA virtual orchestra
• animated musicians playing MIDI encoded music w automatically computed grips
on the instruments w physically based sound
synthesis w sound reverberated according to
virtual concert hall
• music conducted with a baton • neural networks trained to follow the motion • mapping from motion samples to relative timing between
beats • also recognition of conductor’s emotional intent
• performance at Siggraph’97 Electric Garden
![Page 19: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/19.jpg)
eNTERFACE 2013 – Takala 19
Virtual instruments • new user interfaces for synthesized sound
• more degrees of freedom than with a keyboard
• mappings to control parameters of physically based synthesis algorithms
• free configurability è build your own instrument
• examples: w xylophone w drum plate w virtual air guitar (became a media success and later spinned off a game company)
![Page 20: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/20.jpg)
eNTERFACE 2013 – Takala 20
Drawing in the air o Fine motoric interaction methods
for immersive free-hand expression
o A new art medium
o Experiments with artists o exhibition at the Kiasma museum of
modern art (2005) Mäkelä, Reunanen, Takala 2004
![Page 21: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/21.jpg)
eNTERFACE 2013 – Takala
Art installations (student work)
o Experience design o "Tranquil interaction"
21
prisoned dancer
the village
Tommi Ilmonen 2007
![Page 22: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/22.jpg)
eNTERFACE 2013 – Takala 22
Motion-based entertainment and games in VR o hand gestures and sound effects
o virtual aquarium o virtual snow fight
o camera based tracking of full body movements o children's game QuiQui's Giant Bounce o fighting game Kick Ass Kung-Fu
Perttu Hämäläinen
![Page 23: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/23.jpg)
eNTERFACE 2013 – Takala 23
Eyes-free interfaces
• gestural menu selection
• spatialized (3D) auditory display
• use cases when sight is reserved (e.g. while driving) or missing
Raine Kajastila 2013
![Page 24: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/24.jpg)
eNTERFACE 2013 – Takala
4. Current research and trends o Embodied "enactive" interfaces
o Emotions in motion
o Active virtual agents
o Excergames – motion based activities
o Variability and commoditization
o Different thinking in software design
24
![Page 25: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/25.jpg)
eNTERFACE 2013 – Takala
'Enactive media' as research approach o direct and impicit human-machine communication by embodied
activity, without symbolic abstractions inbetween
o aiming at natural interaction with human-like companions o “not using, but living with computers”
o topics: o affective computing (emotions) o virtual agents
25
Meeri Mäkäräinen 1996
![Page 26: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/26.jpg)
eNTERFACE 2013 – Takala
Interaction with captured motion
o analyzing activities and styles by motion descriptors
o animated responsive character programmed with behavioral rules
o may simulate virtual personalities
26
Klaus Förger et al. 2011
![Page 27: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/27.jpg)
eNTERFACE 2013 – Takala
Classification and synthesis of motion
o motion samples observed and characterized by words (human)
o resynthesis by selecting samples from a data base (computer)
o goal: directing animated characters by verbal instructions
27
PCA analysis on motion descriptors
Förger 2013
![Page 28: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/28.jpg)
eNTERFACE 2013 – Takala
o studying psychological/perceptual aspects of games o what affects player's motivation to move?
o presenting captured human motion in a different context
o sample: trampoline as game controller
ExcerGames
28
![Page 29: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/29.jpg)
eNTERFACE 2013 – Takala
Commoditization o basic devices getting cheaper
o webcam o game consoles and controllers (Wii, PSMove, etc)
o large number of special devices available o sensors o interface hw/sw platforms (e.g. Arduino)
o development by hobbyists and open communities
29
![Page 30: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/30.jpg)
eNTERFACE 2013 – Takala
From expensive VR to open platforms
30
RUIS platform
Cave installation http://eve.hut.fi a light-weight cave "Upponurkka"
![Page 31: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/31.jpg)
eNTERFACE 2013 – Takala
Cases o RUIS (Reality-based User Interface System)
o open source platform for affordable VR o see demo on Vimeo
o Using customer level 3DUI for designing o combining Blender modeling software with RUIS UI o demo video: Will it Blend ?
31
![Page 32: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/32.jpg)
eNTERFACE 2013 – Takala
5. Final notice o Open your mind for creativity
o interdisciplinarity, human studies o art and entertainment
o Just do it – be bold, but patient o experiment, prototype, and learn o "The best way to predict the future is to invent it." (Alan Kay)
o KISSS principle in interface design o Keep it simple and stylish, stupid
32
![Page 33: Experimental User Interfaces · 6 eNTERFACE 2013 – Takala not just a notebook, but o multisensory / multimodal o di"erent concurrent devices / kinds of use o image, sound, video…](https://reader033.fdocuments.us/reader033/viewer/2022060421/5f17be845161820b62179e5b/html5/thumbnails/33.jpg)
Thank you!