Slide 1

Slide 2 Multimodality Alistair D N Edwards Department of Computer Science http://www-module.cs.york.ac.uk/advt/classes/class01/materials/Multimodality.pptx Slide 3 But first ADVT Tutorials Lectures Classes Slide 4 ADVT tutorials Autumn term Taken by supervisor Weeks 2, 3, 4, 8, 9, 10 How to write Slide 5 ADVT lectures Alistair Autumn term Week 5 Academic Writing Week Scientific Writing Slide 6 ADVT classes Autumn Term Week 2 Alistair: Multimodality Week 3 Paul: Games Week 4 Helen: Web Accessibility Week 5 Chris: To be announced Slide 7 ADVT classes Spring Term Weeks 2 6 Contents to be announced but last year http://www-module.cs.york.ac.uk/advt/2014-15/index.php See also http://www-module.cs.york.ac.uk/advt/ Slide 8 Multimodality: How many senses do we have? Alistair D N Edwards Department of Computer Science http://www-module.cs.york.ac.uk/advt/classes/class01/materials/Multimodality.pptx Slide 9 Multi-X MULTI-media, MULTI-modal, MULTI-channel... there are many terms that can be prefixed by the qualifier 'multi', so that there is a danger of getting diverted into debates about the meanings of the terms and distracted from the real questions. At the same time, these discussions often centre on computers; a linguist or a playwright or a singer would not even use the terms, far less wonder whether what they do should be prefixed by 'multi'. Slide 10 Multi-X -Why is it that the debate arises in the context of computer interaction? -Why is it that it does not arise in other contexts of communication? -Are there are other contexts in which the 'multi-X' question arises? -Are we progressing towards a situation in which the question will become obsolete or redundant? Slide 11 How many senses do we have? Slide 12 Five Slide 13 Extra senses http://www.bbc.co.uk/radio4/science/extrasenses.shtml 1Pain 2Balance 3Time 4Temperature 5Digestion Slide 14 Class of 2015 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time 16 Pain 17 Fatigue How many senses do we have? Slide 15 Previous answers 1 Sight 2 Smell 3 Taste 4 Hearing 5 Touch 6 Balance 7 Direction 8 Humour 9 Body position 10 Heat 11 Danger 12 Circadian Rhythms 13 Hunger 14 Bodily needs 15 Time/Rhythm 16 Pain 17 Fatigue 18 Pressure 19 Empathy 20 Intuition 21 Fear 22 Hunger 23 Tiredness 24 Thirst Slide 16 What is a sense? Slide 17 What is a human sense? (Lets not get distracted by questions as to whether animals or even plants have senses in the way we mean) Slide 18 What is a human sense? An input channel? Only for external information? or internal too? Slide 19 What is a human sense? An input channel? Only for external information? or internal too? Slide 20 Senses External Internal Compound? A synthesis of inputs fear humour empathy ? Slide 21 Objectives What modes do we use in interaction? Which others could we use? Why? How? Individual senses Combining senses/modes Slide 22 Some practical(?) possibilities Sounds Haptics Smell Slide 23 Sounds Speech synthesized Non-speech earcons auditory icons sonification data mapping audification programs Slide 24 Synthesized speech Who is this? and this? Which is the better voice? How can we make synthetic voices more acceptable? Better use of pauses? Newell, C (2009) Place, authenticity, and time: a framework for liveness in synthetic speech University of York PhD Thesis Newell, C., Edwards, A. D. N. & Cairns, P. (2011) 'Liveness' in human-machine interaction, International Journal of Performance Arts and Digital Media, 7, 2, pp.221-237. Slide 25 Earcons Symbolic sounds with arbitrary mappings to their meanings Paint Open Open paint Slide 26 Earcons Steve Brewster, Glasgow Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1992). A detailed investigation into the effectiveness of earcons. In G. Kramer (Ed.), Auditory display, sonification, audification and auditory interfaces. The Proceedings of the First International Conference on Auditory Display, Santa Fe Institute, Santa Fe, NM: Addison-Wesley, pp. 471-498 Brewster, S. A (1991). Providing a Model for the Use of Sound in User Interfaces, University of York DPhil thesis Slide 27 Algebra earcons Slide 28 Stevens, R.D., Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1994). Providing an audio glance at algebra for blind readers. In G. Kramer & S. Smith (Eds.), Proceedings of ICAD'94, Santa Fe Institute, Santa Fe: Addison-Wesley, pp. 21-30 Stevens, R. D. (1996) Principles for the design of auditory interfaces to present complex information to blind people, University of York DPhil thesis Slide 29 Auditory icon A cartoon sound that resembles the thing it represents Sonic finder Slide 30 Auditory icon Gaver, W. W. (1989). The SonicFinder: An interface that uses auditory icons. Human-Computer Interaction, 4(1), 67- 94 Slide 31 Spearcons Somewhere between speech and an earcon speeded up speech Spearcon 1 Spearcon 1 Spearcon 2 Spearcon 2 Spearcon 3 Spearcon 3 Slide 32 Spearcons Walker, B. N., & Kogan, A. (2009). Spearcons enhance performance and preference for auditory menus on a mobile phone. Invited paper in Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction (UAHCI) at HCI International 2009, San Diego, CA, USA, (19-24 July). Slide 33 Sonification Slide 34 Edwards, A. D. N., Hines G. and Hunt, A. (2008). Segmentation of biological cell images for sonification. 2008 International Congress on Image and Signal Processing (CISP 2008), Sanya, China, IEEE Computer Society, pp. 128-132. Slide 35 Sounds Why do we not make more use of sounds in interfaces? Slide 36 Annoyance? What makes a sound annoying? Slide 37 Experiment Sample 1 Sample 2 Sample 3 Sample 4 Slide 38 Experiment Sample 1 Sample 2 Sample 3 Sample 4 Slide 39 Experiment Sample 1 Sample 2 Sample 3 Sample 4 Slide 40 Experiment Sample 1 Sample 2 Sample 3 Sample 4 Slide 41 Haptics Slide 42 http://www.dcs.gla.ac.uk/~stephen/papers/CHI2000.pdf TermDefinition Proprioceptive Relating to sensory information about the state of the body (including cutaneous, kinesthetic, and vestibular sensations). Haptic Relating to the sense of touch. Vestibular Pertaining to the perception of head position, acceleration, and deceleration Kinesthetic Meaning the feeling of motion. Relating to sensations originating in muscles, tendons and joints Cutaneous Pertaining to the skin itself or the skin as a sense organ. Includes sensation of pressure, temperature, and pain. Tactile Pertaining to the cutaneous sense but more specifically the sensation of pressure rather than temperature or pain. Force Feedback Relating to the mechanical production of information sensed by the human kinesthetic system. Haptics - definitions Slide 43 Geomagic Touch https://youtu.be/nmrviXro65g http://www.sensable.com/ Formerly the Sensable Phantom Omni Slide 44 Invisible 3-D haptics http://phys.org/news/2014-12-team-invisible-d-haptic-video.html Slide 45 Invisible 3-D haptics Carter, T., Seah, S. A., Long, B., Drinkwater, B., & Subramanian, S. (2013, October). Ultrahaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology (pp. 505-514). ACM. Slide 46 Olfactory Slide 47 Trisenx Scent Dome http://www.trisenx.com/ Slide 48 iSmell Slide 49 Slide 50 Olfactory device ode Older people who may forget to eat Releases food smells at meal times Slide 51 ode Fragrances Fresh Orange Juice Cherry Bakewell Tart Homemade Curry Pink Grapefruit Braised Beef Casserole Black Forest Gateau Slide 52 Smells Why do we not make more use of smells in interfaces? Slide 53 Combining senses Slide 54 The bandwidth problem Slide 55 Vision How much information do we receive visually? The real power is in being able to focus Slide 56 Vision versus audition How many screen objects could we hear? ~ 8 Pitt, I. J. and Edwards, A. D. N. (1995). Pointing in an auditory interface for blind users. in Intelligent Systems for the 21st Century: Proceedings of the 1995 IEEE International Conference on Systems, Man and Cybernetics, (Vancouver), IEEE. pp. 280-285. Can use auditory streams Bregman, A, (1990) Auditory Scene Analysis, MIT Press Mitsopoulos, E. N. and Edwards, A. D. N. (1999). A principled design methodology for auditory interactionMitsopoulos, E. N. and Edwards, A. D. N. (1999). A principled design methodology for auditory interaction. in M. A. Sasse and C. Johnson (eds.), Proceedings of Interact 99, (Edinburgh), IOS Press. pp. 263-271 Interact 99, Slide 57 Combining senses Can the whole be greater than the sum of the parts? Slide 58 T4: turning talking tactile tablet Touch Speech Non-speech Rotation haptics (Vision) https://youtu.be/4nW9-zKmKUw Slide 59 Conflicting senses McGurk Effect https://youtu.be/G-lN8vWm3m0 Slide 60 Visual dominance Psychology Neurology User interfaces Slide 61 Conflicting senses Poppadoms Slide 62 Experiment Sample 1 Crisp poppadom, crisp sounds Sample 2 Soft poppadom, crisp sounds Sample 3 Crisp poppadom, white noise Sample 4 Soft poppadom, white noise Slide 63 Conclusions Why do we not make more use of other senses/channels/modes in interaction? Which ones could we practically use more? What would be the benefits? Slide 64 Visual veracity Slide 65