8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
1/62
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
2/62
Research questions
In case of user having multiple devices:
1 Whether they would prefer to switch between multiple device to complete a particular task? IfYes or No then Why?
On what factors this switching depends:
• Type of tasks• Single or Multiple User• Environment
2 Whether they prefer to switch between modalities while completing a particular task? If Yes oNo then Why?
3 Is there any interaction between device and modality preference?
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
3/62
Plan of action
Single User Multiple (Two) Users
Home
Car
Single user inHome context
Two users inHome context
Two users inCar context
Single user inCar context
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
4/62
Plan of action
Single User Multiple (Two) Users
Home
Car
Single user inHome context
Two users inHome context
Two users inCar context
Single user inCar context
11
10
9
9
23
18
Completed
To Be Completed
1
1
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
5/62
Single user in Home context > Scenario and Tasks
You are watching a movie on TV and along with that you are following a Cricket match. Duringcommercial, you switch to Sports channel. You find that one of the player is leading the team to
win the match. You want to post about this on Facebook to show your happiness.
Subtasks :1. Changing the channel
2. Posting on Facebook
(i) Specify the action, (ii) Add text you want to post, (iii) Confirmation
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
6/62
Single user in Home context > Set of Devices
1) TV and smartphone 2) TV, smartphoneand smartwatch
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
7/62
Single user in Home context > Interaction Methods
Smartphone Smartwatch
Touch
Voice
Physical Motion/Gesture
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
8/62
Single user in Home context > Activity
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
9/62
Single user in Home context > Activity
1. First participant is introduced with scenario, subtasks and interaction methods along with anexample
2. He is asked to give first preference for each subtask and then, complete sequence of preferenfor each task.
3. These preferences are taken for both set of devices (TV, Phone; TV, Phone and Watch).
4. While having preferences they are also asked to think aloud to capture the reason behind the
preferences. Therefore, sessions are audio recorded.
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
10/62
Single user in Home context > Activity
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
11/62
You are driving your car to Home and receive a message from a friend saying that your close comfriend is in the town. In message (s)he also invites you at his/her place. You replies back that you reaching there in some time. You opens the navigator and set the destination to friend’s place.
User actions or point of decisions :
Message Notification
1. Your command to read it
System shows/tells you the message
2. Reply: Compose & Send
3. Open Navigator
4. You command system to change the destination to friend’s place
System changes it.
Single user in Car context > Scenario and Tasks
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
12/62
Car Dashboard Smartphone Smartwatch
Touch
Voice
Physical Mot
Single user in Car context > Interaction Methods
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
13/62
2) Car dashboard, Smartphoneand smartwatch
1) Car Dashboard
Single user in Car context > Set of Devices
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
14/62
Analysis
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
15/62
Analysis
1. Device Sets2. Tasks – Type and activity group3. Contexts- Home or Car4. No. of users- One or Two
1. Modality: Preference and Switchi2. Device: Preference and Switching
Effect of changing these On these
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
16/62
1. Tasks that involved changing the value under a
certain variable- Ta
2. Tasks that involved just one button touch/command/input- Tb
3. Tasks that involved composing - Tc
Analysis > Categorization of tasks
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
17/62
Results Summary
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
18/62
In Home context, Gear is preferred more than phone and in Car context, cwas most preferred, when an option among the devices is given
Voice was more preferred over any kind of device generally but touch and gesture wpreferred for particular type of task (ex. Navigation and changing channel).
For a particular task, while changing device sets, modality preference remained mosthe same implying that modality was more important w.r.t. the task than the
Result Summary - Preference
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
19/62
While going from one task to the next, modality switching is more frequentswitching (for both home and car) implying that users try to avoid device cha
than modality change.
Device and modality change is observed more while switching tasks of diffrather than tasks among the same activity.
There is more device and modality switching in case of home than car scen
Result Summary - Switching
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
20/62
Home ScenarioANALYSIS
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
21/62
1. Tasks that involved changing the value under
a certain variable- Ta
2. Tasks that involved just one button touch/command/input- Tb
3. Tasks that involved composing - Tc
T1 > T2
Ta Tb
T1- Changing t
Posting on FacT2: (i)Specify t
T3: (ii) Add texT4: (iii) Confir
Analysis > Categorization of tasks
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
22/62
Combining the tasks w.r.t the activity they belong to
Here,T2, T3,T4 are tasks under the same activity of posting onFacebook.
T1 > T2
Ta Tb
T1- Changing t
Posting on FacT2: (i)Specify t
T3: (ii) Add texT4: (iii) Confir
Analysis > Categorization of sequence of tasks
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
23/62
Home ScenarioRESULTS
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
24/62
Results > Home- Phone
Table 2 PM PT PV
T1 25 20 22T2 14 19 32T3 12 24 30T4 13 24 29
25
1412
13
2019
24 2422
3230
29
0
5
10
15
20
25
30
35
T1 T2 T3 T4
PM
PT
PV
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
25/62
1. Task Wise (Home-Phone):
T1: Modality : Motion >Voice >Touch
T2: Modality : Voice >Touch >MotionT3: Modality : Voice >Touch >MotionT4: Modality : Voice >Touch >Motion
T1- Changing
Posting on FaT2: (i)Specify
T3: (ii) Add tT4: (iii) Conf
Results > Modality preference
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
26/62
TABLE 1
T1 T2 T3 T4PM 32 23 14 21PT 32 40 48 41PV 35 55 56 49GG 51 23 21 28GT 34 34 32 39GV 47 56 60 53
32
23
14
21
32
40
48
41
35
55 56
4951
2321
28
34 3432
3
47
5660
0
10
20
30
40
50
60
70
T1 T2 T3 T4
Results > Home- Phone, Watch
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
27/62
2.Task-wise (Home-Phone, watch):
T1: Device: Gear >Phone
Modality: Gesture >Voice >Touch
*preferences among modalities and devices was very close
T2: Device: Gear >PhoneModality: Voice >Touch >Gesture/Motion
T3: Device: Gear >PhoneModality: Voice >Touch >Gesture/Motion
T4: Device: Gear >PhoneModality: Voice >Touch >Gesture/Motion
T1- Changing
Posting on FaT2: (i)Specify
T3: (ii) Add tT4: (iii) Conf
Results > Device and Mobile Preference
l f
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
28/62
3. Among tasks that were under the same activity (like T2, T3, T4):similar interactions (either in terms of devices or modalities) wereobserved.
Reasons:a) According to accessibility of the device and proximityb) Why was voice more preferred?
lesser chances of distraction from the primary task was involvedin this modality
4. Gear is preferred more than phone when an option between thedevices is given.Although the device is changed, modality preference in doing thetasks remained the same (among the device sets).
Results > Inferences
R l D i i M d li f
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
29/62
5. (a) Task Wise (Home-Phone, Watch ):
T1: Phone: Voice >Touch = Motion
Gear: Gesture > Voice > Touch
T2: Phone : Voice >Touch >MotionGear: Voice >Touch >Motion/Gesture
T3: Phone: Modality : Voice >Touch >MotionGear: Voice >Touch >Motion/Gesture
T4: Phone : Voice >Touch >MotionGear: Voice >Touch >Motion/Gesture
T1- Changing
Posting on FaT2: (i)Specify
T3: (ii) Add tT4: (iii) Conf
Results > Device wise : Modality preference
R lt > D i i M d lit f
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
30/62
5. (b)Independent of tasks:
Phone: Voice >Touch > Motion
Gear: Voice >Touch > Motion/ Gesture* T1: Gesture > Voice > Touch T1- Changing
Posting on FaT2: (i)Specify
T3: (ii) Add t
T4: (iii) Conf
Results > Device wise : Modality preference
R lt > D i d M d lit S it hi g
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
31/62
6. (a) Home- Phone, Watch
(b) Home- Phone
Device and modality change is observed more while switchingtasks of different activities rather than tasks among the sameactivity.
Modality switching is more frequent than device change.
T1- Changing
Posting on FaT2: (i)SpecifyT3: (ii) Add tT4: (iii) Conf
Results > Device and Modality Switching
Device 4 1 0
Modality 8 3 3
Modality 7 4 4
T1 > T2 T2 > T3 T3 > T4
Results > Device and Modality Switching
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
32/62
7. Phone > > Phone + Watch
T1: Modality Switch: 3/11Device Switch : 9/11
T2: Modality Switch: 0/11Device Switch : 7/11
T3: Modality Switch: 1/11Device Switch : 8/11
T4: Modality Switch: 0/11Device Switch : 8/11
Modality preference is more affected by task type than the device.
Results > Device and Modality Switchingbetween device sets
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
33/62
Car ScenarioANALYSIS
Analysis > Categorization of tasks
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
34/62
1. Tasks that involved changing the value under
a certain variable- Ta
2. Tasks that involved just one button touch/command/input- Tb
3. Tasks that involved composing - Tc
T1 > T2
Tb Tc
T1- Your comm
T2- Reply: Com
T3- Open Navi
T4- You comm
change the de
place
Analysis > Categorization of tasks
Analysis > Categorization of sequence of tasks
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
35/62
Combining the tasks w.r.t the activity they belong to
Here,T1 and T2 tasks are under the same activity ofmessaging (notification and composing)
T3 and T4 are tasks under the same activity ofnavigation.
T1 > T2
Tb Tc
T1- Your comm
T2- Reply: Com
T3- Open Navi
T4- You comm
change the
friend’s pl
Analysis > Categorization of sequence of tasks
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
36/62
Car ScenarioRESULTS
Results > Car- Car Dashboard
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
37/62
Results > Car Car Dashboard
21
11
16 16
12
19 1920
27
30
2524
0
5
10
15
20
25
30
35
T1 T2 T3 T4
CS
CT
CV
Results > Car- Car Dashboard Phone Watch
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
38/62
Results > Car Car Dashboard, Phone, Watch
28
3
17
5
18
14
28
38
41
47
43
39
01
0 0
7
13
6
27
16
32
20 20
6
24
1
6
2
7
2
28
36
25
0
5
10
15
20
25
30
35
40
45
50
T1 T2 T3 T4
Results > Inferences
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
39/62
1.Task-wise (Car-Car Dashboard, Phone, watch):
T1: Device: Car > Gear > PhoneModality: Voice >Touch > Gesture
*preferences among modalities and devices was very close
T2: Device: Car > Gear > PhoneModality: Voice >Touch >Gesture/Motion
T3: Device: Car > Gear > PhoneModality: Voice >Touch >Gesture/Motion
T4: Device: Car > Phone > GearModality: Touch > Voice > Gesture/Motion
T1- Your com
T2- Reply: Co
T3- Open Nav
T4- You comm
change th
friend’s
Results Inferences
Results > Inferences
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
40/62
2.Task-wise (Car-Car Dashboard):
T1: Modality: Voice > Steering > TouchT2: Modality: Voice > Touch > Steering
T3: Modality: Voice > Touch > SteeringT4: Modality: Voice > Touch > Steering T1- Your com
T2- Reply: Co
T3- Open Nav
T4- You comm
change th
friend’s
Results Inferences
Results > Device wise : Modality preference
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
41/62
3. (a) Task Wise (Car- Car Dashboard, Phone, Watch ):
T1: Car: Voice > Steering > TouchPhone: Voice >TouchGear: Voice > Touch = Gesture
T2: Car: Voice > Touch > SteeringPhone: Voice >TouchGear: Voice > Touch = Gesture
T3: Car: Voice > Touch > SteeringPhone: Voice >TouchGear: Voice > Touch > Gesture
T4: Car: Voice = Touch > SteeringPhone: Touch > VoiceGear: Voice > Touch > Gesture
y p
T1- Your com
T2- Reply: Co
T3- Open Nav
T4- You comm
change th
friend’s
Results > Device wise : Modality preference
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
42/62
3. (b)Independent of tasks:
Car: Voice > Touch > Steering*T1: Voice > Steering > Touch
Phone: Voice >Touch > Motion*T4: Touch > Voice > Motion
Gear: Voice >> Touch > Motion/ Gesture
* General: This information should be considered for any scenarioirrespective of the experiment done.
y p
T1- Your com
T2- Reply: Co
T3- Open Nav
T4- You comm
change th
friend’s
Results > Device and Modality Switching
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
43/62
7. Car Dashboard > > Car Dashboard + Phone + Watch
T1: Modality Switch: 0/10Device Switch : 3/10
T2: Modality Switch: 0/10Device Switch : 3/10
T3: Modality Switch: 0/10
Device Switch : 2/10
T4: Modality Switch: 0/10Device Switch : 2/10
There was NO modality switching observed between the device sets.
y gbetween device sets
T1- Your com
T2- Reply: Co
T3- Open Nav
T4- You comm
change th
friend’s
Results > Device and Modality Switching
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
44/62
7. Car Dashboard > > Car Dashboard + Phone + Watch
y gbetween device sets
70% 20% 10%
No Modality Switching
No Device Switching
No Modality Switching
Device switching betweensets but continuity throughtasks
No Modality Switching
Device switch from T1 andT2 between sets and switchback to the same device.
T1 > T
Tb T
T1- Your com
T2- Reply: Co
T3- Open Nav
T4- You comm
change th
friend’s
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
45/62
Home and Car ScenarioRESULTS
Results > Home and car scenario
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
46/62
Modality and device switching
0
5
10
15
Home Car
Device
Modality
Home Car
Device 5 2
Modality 14 9.5
In the cadevice sC+P+W
There isand modin case ocar scen
In both tmodalityhigher tdevice.
Results > Home and car scenario
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
47/62
Modality Preference
Home (Phone)T1 Ta MotionT2 Tb VoiceT3 Tc VoiceT4 Tb Voice
Car (Car Dashboard)Tb VoiceTc VoiceTb Voice
Ta/c Voice
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
48/62
Themes and framework
Themes > Affinity
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
49/62
I n t u i t i v e n e s s D
i s t r a
c t i o
n
E a s e O F U s e E f
f o r t
Learnability
Familiarity
User Control
Continuity PhysicalFeatures
Feedback
Flexibility
Cause-Effect Framework
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
50/62
Home
Cause-Effect Framework
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
51/62
Car
Cause-Effect FrameworkC H
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
52/62
H
H
H
H
H
H
H
H
H
Car -> How to use
Comparison between featuresCar Travel booking
Cause-Effect FrameworkC H
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
53/62
M
L
L
M
M
M
M/L
L
Car -> How to use
Comparison between featuresCar Travel blogging
Cause-Effect FrameworkC > H t
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
54/62
Car -> How to use
Ideating on travelblogging
1. How to lower the informationrequired by the system/user?
2. How to Lower the frequency of interactions?3. How to use the environment?4. How to make the user feel controlled?5. How to increase the flexibility in the way of using a modality?6. How to incorporate the familiarity of a certain device in the feature?7. How to lower the distraction from the feedback?
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
55/62
Themes and inferences
Themes > Inferences
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
56/62
Distraction from the primary task came out to be major reason for voice to be themost preferred modality in both car and home scenario.
Most of participants wanted Facebook to be opened on TV so that they can avoidchanging devices whereas few others wanted it to be in phone because ofassociation of facebook with phone and occlusion etc. - Continuity
As confirmation is the final step , they always wanted to be sure about the abecause if here, they do something wrong, either they will go the initial step orsomething wrong will be posted.- User Control
Users feel more confident and in control while using touch than gesture becausefamiliarity and tangible feedback
Themes > Inferences
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
57/62
While composing text modality preference depends on type and size of text. If it’scomplex or big text they are likely to prefer touch whereas for small or simple textthey would choose voice- Flexibility
While switching channels, modality preference depends on distance b/w the twochannels-> modality preference e.g. just next channel-> motion, going to aparticular channel: voice, browsing: multiple channels- Flexibility
Few participants reported that choice of modality depends on type of feedreceived from system in previous task e.g. If system is communicating throughaudio, user tend to use voice.
Themes > Inferences
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
58/62
Participants found opening Facebook while watching TV too specific and lessfrequent than changing channel so they responded that it is likely to define andremember a gesture/motion for changing channel rather than opening facebookLearnability
Users always tend to avoid gestures and motion when multiple interactions have tobe done for completing same task e.g. browsing through channels. Touch is morepreferred in such cases.- Effort
Themes > Inferences
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
59/62
Varied Perception about voice:
Many people thought it to be quick and natural way of communication. But, fewothers told that they have to be conscious while giving commands, specially, whencommands are small or trivial.
For voice, I have to be more conscious, it won’t be the same way I talk. So it’s not the natural always.
I will use voice as it is natural and intuitive than touch .
Too much of voice inputs would look commanding.
TRIVIAL PART COMES IN PLAY. Voice is involving more effort and more energy, eventually my throatup . Saying same input to myself will sound mad.
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
60/62
Appendix
Home- Phone, WatchT1 PM PT PV GG GT GV
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
61/62
Phone, Watch
T1 PM PT PV GG GT GVPref 1 2 0 0 5 0 4Pref 2 2 1 4 2 0 2Pref 3 0 2 1 1 6 1Pref 4 1 5 2 1 1 1Pref 5 1 1 1 2 3 3Pref 6 5 2 3 0 1 0
32 32 35 51 34 47
T2 PM PT PV GG GT GVPref 1 0 1 3 0 0 7Pref 2 0 3 6 1 1 0Pref 3 2 2 1 1 3 2Pref 4 2 2 1 0 4 2Pref 5 2 2 0 5 2 0Pref 6 5 1 0 4 1 0
23 40 55 23 34 56
T3 PM PT PV GG GT GVPref 1 0 2 1 0 0 8Pref 2 0 1 10 0 0 0Pref 3 0 7 0 0 1 3Pref 4 0 1 0 2 8 0Pref 5 3 0 0 6 2 0Pref 6 8 0 0 3 0 0
14 48 56 21 32 60
T4 PM PT PV GG GT GVPref 1 0 2 1 0 1 7Pref 2 0 2 7 1 1 0Pref 3 1 2 1 2 4 1Pref 4 2 3 1 1 3 1Pref 5 2 1 0 5 1 2Pref 6 7 0 1 2 1 0
21 41 49 28 39 53
Home- Phone
8/18/2019 Exploring Design for Multi-device, Multi-environment and Multimodal Connected Experiences
62/62
Phone
T1 PM PT PV
Pref 1 6 1 5
Pref 2 2 7 1
Pref 3 3 3 5
25 20 22
T2 PM PT PV
Pref 1 0 1 10
Pref 2 3 7 1
Pref 3 8 2 0
14 19 32
T3 PM PT PV
Pref 1 0 3 8
Pref 2 1 7 3
Pref 3 10 1 0
12 24 30
T4 PM PT PV
Pref 1 0 3 8
Pref 2 2 7 2
Pref 3 9 1 1
13 24 29
Top Related