Page 1
Touching More than 3 Dimensions Research Into Novel Interfaces – three dimensions
and above
Joaquim JorgeINESC-ID & IST / U Lisboa
Portugal
IP / Leiria Nov 2015
Page 2
Joaquim JorgeInstituto Superior Técnico
Universidade de LisboaVisualization and Multimodal Interfaces
@ INESC-ID Lisboahttp://web.ist.utl.pt/jorgej
ACM/SIGGRAPH Small Conferences Committee Chair
Editor Computers & GraphicsResearch Interests: Calligraphic Interaction, Multimodal Interfaces, Graphical Modeling
Page 3
IS TONY STARK HOME IS TONY STARK HOME ?
Page 4
WE HAVE SEEN THE FUTURE…
Page 5
Natural Interfaces ?
Page 6
http://youtube.com/watch?v=oIOxT6PUgPA
Dassault Systèmes Vision
Page 7
NATURAL INTERACTION Ivan Sutherland – 52 years ago
Page 8
Ivan Sutherland SketchPad Video
youtube.com/watch?v=495nCzxM9PI
Page 9
Beyond 2D+ Space Continuous Above+ Hands Bimanual Interactions+ Dimensions 3D UIs+ Expressiveness Sound, Kicking+ Proxemics Eery Space
Page 10
INTERACT 2011Interaction Techniques Unifying Touch and Gesture On and Above a Digital Surface
Page 11
The Continuous Interaction Space Video
http://youtu.be/A8kb7BV7j1Y
Page 12
Exploring Bimanual Interaction on a TableTop
www.shapeshop3d.com
Page 13
Pedro Lopes, Daniel Mendes, Bruno Araújo, Joaquim A. JorgeCombining bimanual manipulation andpen-based input for 3D modeling, SBIM 2011
Page 14
Key Points
Table ArchitectureCombining Pen and GesturesAsymetric Guiard Model
Page 15
Table Architecture
Page 16
MOVIE
See Movie at: https://vimeo.com/152554942
Page 17
EvaluationTest Procedure
Free drawing
Page 18
EvaluationMore sketching, less camera
Page 19
Evaluation
Camera operations: more fluid
(seconds)
44% faster
Page 20
EvaluationHand Role Distribution
9% 91%
3% 80% 17%
Page 21
EvaluationUsers prefer touch for all manipulations
Imagem e benefits of direct vs indirect
Page 22
EvaluationPrefer Combined technique
Page 23
ObservationsBimanual workspace and hand role
Page 24
ObservationsNatural resting position of NDH
Page 25
ObservationsPen-only often forces re-arrangement
Page 26
ObservationsPen-only causes too much DH motion
Page 27
ObservationsRotate gesture was frail, swapped pen to NDH
Page 28
ObservationsCombined gestures were naturally used
Page 29
Getting it together: Mockup Builder
Page 30
SetupMultitouch 120HzTouch 16.25 DPI3D SVGA 120HzActive StereoKinect (30Hz)Gametrak (120Hz)
Page 32
User ModelActive User KinectHead Position KinectHand Position (Right/Left) KinectFinger touches Table4 Fingers tracked in Space (Gametrak)
Finger X from Hand Y on or above the surface from user Z
Page 33
BIMANUAL AsymmetricDH : SketchingNDH : Manipulation
Modes Hands (WIMP)
Page 34
CONTINUOUS INTERACTION SPACE
Page 35
NDH: Context Menus DH: Selects
Page 36
ON & ABOVE THE SURFACE
Snapping
Discrete Constraint
Continuous Constraints
On the flyActive as long as needed
Page 37
Preliminary Results
expert architect Rhino3D5’20”
2’45”
5’41”
3’46”
Expert User our modeler
Page 38
Touching The 3D Dimension, Dagstuhl Feb. 2012
Modeling On and Above a Stereoscopic Multitouch Display, (3DCHI): May, 2012.
Mockup Builder: Direct 3D Modeling On and Above the Surface in a Continuous Interaction Space (GI'2012), May 28 - May 30, 2012.
Special Section C&G (January 2013)
ACM Spatial User Interfaces Symposia (SUI)
Page 39
Adding Sound to Multimodal Interfaces
ITS 2011
Page 40
Take-Home message
Touch signature
Impact Intensity
Interaction space
Tangibles
Page 41
Take-Home message
Touch signature
Impact Intensity
Interaction space
Tangibles
Page 42
Take-Home message
Touch signature
Impact Intensity
Interaction space
Tangibles
Page 43
Take-Home message
Touch signature
Impact Intensity
Interaction space
Tangibles
Page 44
Acoustics of Touch
Page 45
Acoustics of Touch
Page 46
Acoustic SensingSetup
Page 47
Interaction Opportunities
https://vimeo.com/30997725
Page 48
https://vimeo.com/30997725
See Video
Page 49
Acoustic SensingConclusions
Affordable Setup
Page 50
Acoustic SensingConclusions
Understand impact (amplitude)
Page 51
Acoustic SensingConclusions
Recognize contact parts (timbre)
Page 52
Acoustic SensingConclusions
Non-tracked Tangibles
Page 53
Acoustic SensingConclusions
Multiplicity of Input
Page 54
Let’s Kick ItCHI 2014
Exploring New Interaction Space
Large Scale Displays
Interacting with the bottom of the screen
Page 55
Beyond Situated Displays
Eery Space
Page 56
Remote Proxemics: Person to Device Interactions
Page 57
Social Interactions
Page 58
Proxemics Distances [Hall, 1966]
Page 60
Providing Awareness: Floor Circles
Page 61
Issues
Working / Virtual SpaceSimple Operators / DialoguesImplicit Mode ChangesContinuous TransitionAsymmetric Bimanual Modes
Page 62
Take Home MessageExtending Touch With Other ModalitiesBimanual InteractionExplore Surface Above TableUse Sound in Novel WaysExplore New ModalitiesProxemics as tool for implicit interactions
Page 64
Content Creation in VEsSketch Interpretation (3D) New ModalitiesSelf Disclosure (Not!)Understandability
Page 65
Thank You!Joaquim Jorge
http://web.ist.utl.pt/jorgej
Bruno Rodrigues De AraújoMaurício de Sousa
Ricardo Jorge Jota CostaPedro Lopes
Daniel Mendes