Multimodal Interaction, Feb 6, 2019
IntroductionWolfgang Hürst
Contact: [email protected], http://people.cs.uu.nl/huerst
1
Contact information
Lectures 1-4, guest lecture & examPeter Werkhoven, UU & [email protected]
Lectures 5-8, exam & projectWolfgang Hürst, UU, Multimedia [email protected], BBG 480(contact me also for administrational questions)
When you email us, start subject with [INFOMMMI]
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 2
IMPORTANT!
Make sure to regularly check your student email, since some important information, last minute
changes, etc. might be sent by email.
It is also a good idea to keep an eye on the “news” page of the course’s website.
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 3
Multimodal interaction
General Introduction
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 4
How are we interacting with computers?
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 5
KEYBOARD, MOUSE
GRAPHICAL USER INTERFACES
(MULTI) TOUCH
SPEECH
GESTURES & MOTION
DIFFERENT DISPLAYS
Computer usage – from data to experience
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 6
1D 3D
Punch cards & cables, …
data information & communication
experience
Touch (& mouse) gestures, …
CLI (Command Line Interface)
Keyboard & mouse
Trackpads, touchscreens, …
Body motions / gestures, tangible UIs, …
Speech, …
Mainframes & workstations PCs & internet Mobile devices,
HMDs, projective displays, …
GUI (Graphical User Interface)
From to to
2D
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 7
This requires• Looking into different technologies• Looking into human aspects
(e.g. perception of visual, tactile, and acoustic feedback)
What modalities make sense?And when? And why?
Goal of this course: Learn about different modalities for interaction and when and how to use them.
Overview(lectures)
Human perception• Hearing• Vision• Touch• Multi-sensory• Synaesthesia
Virtual reality• Applications• Technology• Interaction• Navigation• Mobile VR
Augmented reality• Technology• Human factors• Interaction
Lectures are complemented with papers for mandatory reading.Make sure to read them in time, so you have enough time for the project.
Example for human perception …
2/6/19 9W. Hürst, http://people.cs.uu.nl/huerst
Source: National Geographic – Brain Gameshttp://natgeotv.com/nl/brain-games/videos/rubberen-hand
Integration of visual and tactile stimuli in a stereoscopic virtual environment
Goal: investigate perception of incongruencies in location between tactile and visual stimuli
Lecture Project Recommended actions
Week 1Introduction
Papers 1, 5
Week 2 Lecture 1 (vision)Lecture 2 (sound, …) Deadline for forming groups
Papers 2, 3, 4 & forming project groups
Week 3 Lecture 3 (VR)Lecture 4 (other)
Papers 6, 7, 8 & work on project ideas, prepare pitches, plan
Week 4 Project pitchesGuest lecture
Deadline pitch slides: Feb 25, 9:00Deadline project plan: Feb 27, 23:59
Pitch ideas & make project plan
Week 5 Lecture 5 (AR)Lecture 6 (AR)
AR papers &setup and implementation
Week 6 Project presentationLecture 7 (AR)
Deadline slides: Mar 11, 9:00 AR papers &setup and implementation
Week 7 Lecture 8 (AR) Experiments
Week 8 Analysis & report / paper writing
Week 9Project presentation(final) & meetings
Deadline project report: Apr 1, 23:59
Examweek
Exam
GradingGRADING PASSING
EXAM (lectures 1-4, guest, lectures 5-8)
Exam grade counts 50% for final grade
Min. 5.0 (before rounding)
PROJECT Project grade counts 50% for final grade
Min. 5.0(before rounding)Min. 5.5 (before rounding)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 12
Participation in all grading-related events (exam, project presentations & meetings) is mandatory.
• Participation in all events (exams, project presentations and meetings)
• Final grade at least 4.0 (after rounding)
Retake: requirements for qualification
Passing
Note: Attendance of the lectures is mandatory, too!
However, we are not actively checking it (but expect exam questions that are not covered by the slides and literature).
Attendance in the meetings and presentations related to the project is mandatory and will be checked.
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 13
Exceptions may be made if you have a time conflict (but contact me before the event).
• If you failed because your exam average or your overall grade was too low, you can do the retake exam
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 14
• If you failed because your grade in the project was too low, you can get a new assignment
Keep in mind that the retake exam is after the 4th quarter (consider this when planning your summer vacation)
Retake – Requirements for passing
• Your retake grade (exam and/or project) will replace the corresponding grade (exam and/or project).
• You pass this course if your new grade fulfills the original requirements
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 15
Multimodal interaction
Project IntroductionGeneral remarks
Why a project?
• Hands-on experience is important (esp. when dealing with interaction)
• Many of the learning goals of this projectare motivated by lacks of skills that I identified in small and thesis projects
• Mostly self-dependent work, esp. with respect to
• Coming up with research ideas & solutions• Specifying your research goals & questions
• Time planning & scheduling
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 16
Why a group project?
• Notice the difference between “working self-dependent” and “working alone”
• Lack of self-dependence is also about bringing in own ideas, discuss and defend them, dealing with rather vague specifications
• Important skills for team work, which will likely be part of your job (and your thesis project, which usually contributes to a bigger research effort)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 17
How to build the groups?
• Size: 4-5 members should be ideal (but 3-6 may be okay, too, after discussion)
• You are free to chose your team mates (assignment by me only if necessary)
• Keep in mind your skills and experience when building your teams and specifying your goals (e.g., time is too short to learn a new programming language from scratch)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 18
Use the spreadsheet at http://tinyurl.com/infommmi2019to sign up for a group and find team members.
Project
Topic & taskWhat you can/have to do
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 19
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 20
In the lectures, you will learn a lot about different modalities and how we perceive them.
And how we can use technology to create multimodal interaction experiences or how multimodality affects interaction.
Based on this, you should come up with your own idea of a small multimodal interaction research project.
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 21
In general, your project will have three phases / parts:1. Setup / implementation2. Experiment3. Analysis
Notice that actual development is preferable but not essential.
Topic & tasks for project
Restrictions:• You are not allowed to hit people with a hammer• No humans or animals should be harmed• No laws should be broken
Any other multimodal experiment should be fine J
Examples for multimodal / multisensory experiments
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 22
Localization (small project)
Tactile type (MSc thesis)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 23
Parinya Punpongsanon, Daisuke Iwai, Kosuke Sato. SoftAR: Visually Manipulating Haptic Softness Perception in Spatial Augmented Reality. IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE International Symposium on Mixed and Augmented Reality 2015), Vol. 21, No. 11, PP. 1279-1288.
Examples for multimodal/multisensory experiments
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 24
Sungjae Hwang, John Song, and Junghyeon Gim. 2015. Harmonious Haptics: Enhanced Tactile Feedback Using a Mobile and a Wearable Device. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15).
Examples for multimodal/multisensory experiments
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 25
Examples for multimodal/multisensory experiments
If you have a smartphone & smartwatch (with vibration motors), you could do a similar project
Interesting questions to study:• Different vibration patterns (see paper)• Evaluate the influence of location
(wrist vs. tablet vs. both)
If you use Google Cardboard (and a 2nd
smartphone), you could even do this in VR
You could also use a smartphone for handheld AR (and test, e.g., different feedback such as visuals vs. sound vs. tactile)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 26
Examples for multimodal/multisensory experiments
Other simple setups include• Visuals & tactile
(e.g., can you influence tactile perception with visuals?)
• Visuals & sound (also on PCs)
Note:• You don’t have to use mobiles,
but can use any tools you like.• You don’t have to do VR/AR,
but anything involving multimodality is ok.• You don’t have to use multiple modalities,
but a test with one modality is also fine, if it fits to the course’s topic
The major criteria is that you demonstrate that you learned something (useful) from the course.
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 27
In previous years, one group investigated the impact of airflow on a roller coaster VR experience.
Notice that this is an example for an experiment that did not require any actual development.
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 28
One group tested the influence of audio (e.g., scary vs. happy background music) in a simple VR environment (floating cubes).
Notice that this example required some rather simple development (i.e., a well-suited topic for beginners)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 29
One team used vibration motors hooked up to an Arduino to study frictional effects in VR and their impact on presence
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 30
And one group put various sensors (distance measures, tactile and auditory feedback) into a head to provide navigation aid for visually impaired.
Evaluation in a setup maze environment with blindfolded participants and combinations of none, auditory, and tactile feedback.
Usage of tools (hardware & software)
You are welcome to use any hardware you have available (but make sure to guarantee availability during the whole project)
You are welcome to use existing software (but make sure to specify it in your project report)
We can provide you with: • HTC Vive VR HMDs (max. 3 are available)• Two old Oculus Rift VR HMDs (1 DK1 and 1 DK2)• Meta 1 and Meta 2 AR HMDs
Note that you might need to share them with some MSc students and there’s a lack of computers and rooms (sorry for that).
You are encouraged to take advantage of this, but not required (e.g., non-VR projects are fine, too).
We will discuss hardware assignments and availability after your pitches.
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 31
Lecture Project Recommended actions
Week 1Introduction
Papers 1, 5
Week 2 Lecture 1 (vision)Lecture 2 (sound, haptics) Deadline for forming groups
Papers 2, 3, 4 & forming project groups
Week 3 Lecture 3 (VR)Lecture 4 (other)
Papers 6, 7, 8 & work on project ideas, prepare pitches, plan
Week 4 Project pitchesGuest lecture
Deadline pitch slides: Feb 25, 9:00Deadline proj. plan: Feb 27, 23:59
Pitch ideas & make project plan
Week 5 Lecture 5 (AR)Lecture 6 (AR)
AR papers &setup and implementation
Week 6 Project presentationLecture 7 (AR)
Deadline slides: Mar 11, 9:00 AR papers &setup and implementation
Week 7 Lecture 8 (AR) Experiments
Week 8 Analysis & report / paper writing
Week 9Project presentation(final) & meetings
Deadline proj. report: Apr 1, 23:59
Examweek
Exam
Complaint in a previous year:
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 33
“This is way too little time for a project like this!”
Complaint in a previous year:
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 34
“This is way too little time for a project like this!”
• The course is 7.5 ECTS, which corresponds to 210 hours.
• The project grade counts 50%, so 105 hours.
• Multiply this by number of team members(and subtract some hours for communication overhead).
With good planning, this should be enough to do something great!
Complaint in a previous year:
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 35
“This is way too little time for a project like this!”
Complaint in a previous year:
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 36
“This is way too little time for a project like this!”
• You decide what you do.• It is part of the learning goals to:
– Find an interesting & relevant topic– Find a feasible topic!– Make a good time planning!– And stick to it!
Good planning & topic selection
• It is okay when your study has limitations …… if you can explain them and are aware of the consequences.
• It is okay to (partly) fail because you aimed too high …… if you can explain why and still achieved a learning goal
(and your original goal was realistic at the time)
• Important issues are:– That you demonstrated that you learned something (useful)
from the course– That your plans and study make sense as a whole (even if they
have limitations)
Note: you are allowed to take risks, be creative, go crazy, surprise me with cool ideas!
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 37
You can approach this from two ways:• Study a cognitive issue
(but don’t forget to address the practical impact)• Look at an applied/computer engineering problem
(but make sure to address a scientific problem)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 38
But how to get from an ideato a research questionto scientific results?
Project
How to come up with a good research topic?Some general tips & comments
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 39
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke Johnson, Larry Christensen, Fig. 3.1, page 64
Research problemThe issue or problem within a broad topic area
Research purposeA statement of the intent or objective of the study
HypothesisA prediction or best guess of the relation that exists among the variables
being investigated
Research questionIn quantitative research it is an interrogative sentence that asks a question
about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue,
or phenomenon to be explored.
Research topicThe broad subject matter area to be investigated
For self study
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke Johnson, Larry Christensen, Fig. 3.1, page 64
Research problemThe issue or problem within a broad topic area
Research purposeA statement of the intent or objective of the study
HypothesisA prediction or best guess of the relation that exists among the variables
being investigated
Research questionIn quantitative research it is an interrogative sentence that asks a question
about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue,
or phenomenon to be explored.
Research topicThe broad subject matter area to be investigated
Notice that this is from “Educational research” (but it applies to all kinds of empirical and quantitative research)
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topic Notice the differences:• From the general context / bigger aim• To a concrete research (sub) problem• To a very concrete research questions
that can be answered with scientific methods
Example:• MSc thesis
Nina Rosa (published at ICMI’15)
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topicResearch topic:The broad subject matter area to be investigated
Example:We know from literature that incongruent stimuli can change perception. Can we take advantage of such effects to create richer VR experiences by providing incongruent tactile and visual stimuli?
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topic Research problemThe issue or problem within a broad topic area
Example:How does perception of tactile feedback change when incongruent visual stimuli are presented simultaneously?In particular, can we use certain visual signals to create a certain tactile perception?
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topicResearch purposeA statement of the intent or objective of the study
Example:Investigate if visual stimuli that represent certain characteristics or experiences can be used to modify our perception of certain tactile stimuli accordingly.
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topic Research questionIn quantitative research it is an interrogative sentence that asks a question about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue, or phenomenon to be explored.
Example:Is it possible to create the illusion of experiencingdifferent intensities of a certain property (weight or temperature) using a rather simple and unrelated type of touch (vibrations) together with compelling, type-related visuals (speed/size and color, respectively)?
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topic Terminology:• Independent variables = the input or
cause (i.e., what we control)• Dependent variables = the output or
outcome whose variation is being studied (i.e., what we measure)
The purpose of an experiment is generally to verify how the latter depend on the former.Ideally, they are reflected in your research question.
Example:• Independent variables:
• Color (red, gray, blue)• Falling speed and size
• Dependent variable:• Experienced intensity of properties
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topicHypothesisA prediction or best guess of the relation that exists among the variables being investigated
Example:• We expect that visuals commonly
associated with weight (speed/size) lead to an increased perception of tactile intensity
• We expect that visuals commonly associated with temperature (color) lead to an increased perception of tactile intensity
But what if we can’t make such an educated guess?
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topicHypothesisA prediction or best guess of the relation that exists among the variables being investigated
Terminology• Null-hypothesis: assumes no
relationship or significant difference between groups on one variable
• Directional hypothesis: predicts an outcome in one direction based on prior literature or other reasoning
• Nondirectional hypothesis: similar to directional hypothesis, but the exact form of difference (e.g., higher, lower, more, less) is not specified
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topic
Be careful! There is no general agreement on the terms on the left!
People use the same term in different ways.
But the basic procedure (from “high level/big picture” to “concretely verifiable statement”) is generally the same.
The development of a research idea
Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke
Johnson, Larry Christensen, Fig. 3.1, page 64
Research problem
Research purpose
Hypothesis
Research question
Research topic Global aim / general goal
Related research problem(s) that need to be solved to achieve this
(big) goal
Related research questions that need to be answered to solve these
problems
Concrete objectives that need to be addressed to answer them
This is the terminology that I normally use.
The development of a research idea
Global aim / general goal
Related research problem(s) that need to be solved to achieve this
(big) goal
Related research questions that need to be answered to solve these
problems
Concrete objectives that need to be addressed to answer them
This is the terminology that I normally use.
And don’t forget that the sub-problem
that you address has to
be relevant!
For your project
2/6/19 53
What (general) problem are you approaching?What concrete sub-problem will you address?How are you going to verify your ideas/solutions?
Three slides (strict; plus one intro & one closing).
1. Problem (general goal and concrete sub problem)
2. Solution (your idea to address the sub problem and/or the research question you want to answer, related hypothesis, and objectives)
3. Methodology (your planned approach to verify your results or proof that your solution works)
Time limit is 5 minutes (& will be strictly enforced).
Recommendation for brainstorming: Try answering these questions
Project pitch (Feb 25):
Project pitches1. Problem2. Solution3. Methodology
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 54
Note: your problem and solution specification depend on your goal.
Most of you will be doing some sort of an empirical evaluation.
It is okay though to address a more engineering problem or an HCI-related issue.
While they use different approaches, they usually follow the same ”top-down” structure illustrated before.
And they also rely on objective evaluations (which generally means your problem solution should be measurable and results should be repeatable).
Project
Scientific researchSome terminology and info
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 55
Note: also pay close attention to the methodology used in the
papers for mandatory reading and learn from them.
For self study
Using SMART criteria for setting objectives
• Specific – target a specific area for improvement.
• Measurable – quantify or at least suggest an indicator of progress.
• Assignable – specify who will do it.
• Realistic – state what results can realistically be achieved, given available resources.
• Time-related – specify when the result(s) can be achieved.
Generally applied to project management, employee-performance management and personal development, but used (for example) in research proposals as well.
Helpful for your course project (and later small project & master thesis)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 56
See, e.g., https://en.wikipedia.org/wiki/SMART_criteria
You
See planning & limitations discussed before
These are also characteristic for research projects!
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 57
Using SMART criteria for setting objectives
• Specific – target a specific area for improvement.
• Measurable – quantify or at least suggest an indicator of progress.
• Assignable – specify who will do it.
• Realistic – state what results can realistically be achieved, given available resources.
• Time-related – specify when the result(s) can be achieved.
Be careful: the term “objectives” is used in different contexts, e.g.:• High level / general goal, e.g.,
Create a SW Kit for a tactile VR suit• Specific / concrete subtest, e.g.,
Specify tactile resolution on lower arm under varying conditions
Notice though what both have in common: specific & measurable
2/6/19 58
Source: “What makes a good research question?” http://twp.duke.edu/uploads/media_items/research-questions.original.pdf(Note: this is from humanities, where people are often less specific about the 2nd issue, i.e., “measureable”)
Examples for “specific”
1. Too broad and does not define the segments of the analysis:Why did the chicken cross the road?(The question does not address which chicken or which road.)
2. Too specific (could be answered with simple internet search):How many chickens crossed Broad Street in Durham, NC, on Feb 6, 2014?(Ostensibly, this question could be answered in one sentence and does not leave room for analysis. It could, however, become data for a larger argument.)
3. A more precise question might be the following:What are some of the environmental factors that occurred in Durham, NC between January and February 2014 that would cause chickens to cross Broad Street?(This question can lead to the author taking a stand on which factors are significant, and allows the writer to argue to what degree the results are beneficial or detrimental.)
Scientific method: Generally an iterative process ofa) observation & b) experimentation
Generally done by measuring data under varying conditions (with the goal to answer some research question, i.e., gain relevant scientific knowledge)
What are some of the environmental factors that occurred in Durham, NC between January and February 2014 that would cause chickens to cross Broad Street?
”Relevant knowledge”, i.e.:• Is the result of general value (bigger context & contribution)?• Is the result applicable to a general scenario (other conditions, etc.)?
Þ It is also important to be able to analyze the data with respect to the research question & goals
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 59
Measuring data under varying conditions (with the goal to answer some research question, i.e., gain relevant scientific knowledge)
What data to measure (& how)?
Quantitative, e.g., values that can be measured and expressed numerically• Different types exist (discrete/continuous, …)• Usually analyzed with statistical means
(averages, distributions, significance testing)
Qualitative, e.g., comments & observations• Can be subjective & biased• Not directly quantifiable, but means to measure exist, too,
e.g., categorize and encode for gaining general results
Note:• Separation not always strict (e.g., quantitative description of qualitative data
via questionnaires)• It can make sense to gather both (e.g., qualitative data to gather insight into
quantitative results, comparison of perceived versus actual performance, etc.)2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 60
How to gather qualitative data?
• Questionnaires• Formal or structured interviews• Cognitive walkthroughs (esp. in HCI)• Etc.
Example: user engagement
”User engagement is the emotional, cognitive, and behavioral connection that exits, at any point in time and possibly over time, between a user and a resource.”
From Attfield et al., 2011: Towards a science of user engagement
(position paper)
Research trend (esp. in HCI; in addition to usability)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 61
Example: user engagement
How to measure?
Self-report:What: Happy, sad, enjoyment, …Means: Questionnaire, interview, think-aloud and think after
protocols, ...Attributes: Subjective, short- and long-term, lab & field, small scale
Physiology:What: Gaze, body heat, mouse movement, ...Means: EEG, SCL, fMRI, eye tracking, mouse-tracking, ...Attributes: Objective, short-term, lab & field, small & large scale
Analytics:What: Click, upload, read, comment, share, ...Means: Intra and inter-session metrics, data science, ...Attributes: Objective, short- and long-term, field, large scale
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 62
Example: user engagement
How to measure?
Self-report:What: Happy, sad, enjoyment, …Means: Questionnaire, interview, think-aloud and think after
protocols, ...Attributes: Subjective, short- and long-term, lab & field, small scale
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 63
Þ Qualitative measures
Established questionnaires often used in this context include:• Focused attention questionnaire [O’Brien & Toms, 2010]• PANAS questionnaire [Watson, Clark & Tellegen, 1988]
Example: user engagement
Focused attention questionnaire by O’Brien & Toms (2010)
1. I lost myself in this news tasks experience
2. I was so involved in my news tasks that I lost track of time
3. I blocked things out around me when I was completing the news tasks
4. When I was performing these news tasks, I lost track of the world around me
5. The time I spent performing these news tasks just slipped away
6. I was absorbed in my news tasks
7. During the news tasks experince I let myself go
Ratings on a 5-point Likert-scale from “strong disagree” to “strong agree”
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 64
Example: user engagement
PANAS questionnaire by Watson, Clark & Tellegen (1988)
“You feel this way right now, that is, at the present moment?”
Ratings on a 5-point Likert-scale with
1 = very slightly or not at all2 = a little3 = moderate4 = quite a bit5 = extremely
for the following 10 postive and 10 negative items (presented in randomized order):
• Distressed, upset, guilty, scared, hostile, irritable, ashamed, nervous, jittery, afraid
• Interested, excited, strong, enthusiastic, proud, alert, inspired, determined, attentive, active
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 65
Example: user engagement
PANAS questionnaire by Watson, Clark & Tellegen (1988)
Comments:
• Answers are used to calculate a mean positive affective score and a mean negative affective score (by summing up ratings for positive and negative items).
• The original paper verified the reliability and validity of this approach
• It provided a ground truth, i.e. for a “normal population”, the mean positive affective score should be 29.7 (SD = 7.9) and the negative affective score should be 14.8 (SD = 5.4)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 66
General comments (not just for qualitative data):
• Be careful that you do not just end up gathering data
• Make sure you can gain some meaning and knowledge out of it!
For example:
• With no ground truth, what to compare against?
• If you have multiple conditions, are people able to distinguish them at the end of a long test session?
But if you ask them in between conditions, won’t that influence their later answers?
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 67
Measuring data under varying conditions (with the goal to answer some research question, i.e., gain relevant scientific knowledge)
What data to measure (& how)?
Quantitative, e.g., values that can be measured and expressed numerically• Different types exist (discrete/continuous, …)• Usually analyzed with statistical means
(averages, distributions, significance testing)
Qualitative, e.g., comments & observations• Can be subjective & biased• Not directly quantifiable, but means exist to,
e.g., categorize and encode for gaining general results
Note:• Separation not always strict (e.g., quantitative description of qualitative data
via questionnaires)• It can make sense to gather both (e.g., qualitative data to gather insight into
quantitative results, comparison of perceived versus actual performance, etc.)2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 68
How to gather quantitative data?• Logging, recording, other measurements, …
The type of data usually suggets how, but be careful what you record, e.g., common mistakes:• Rounding errors, delays, ...• Do you have the right data?
(E.g., comparing averages requires comparable sample sizes)
What type you need to record depends on your research question, e.g.:• Independent variables (what you control)• Dependent variables (what you measure)Also: consider confounding factorsAnd: recording other data might help to gain further insight
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 69
Confounding factors
• Variables other than the independent variable that might affect the dependent variable
• Related to potential issues of your studies
(Note: they likely always exist, but try to minimize them and their potential influence)
Verify your experiment design with respect to:
• Reliability (the extend to which an experiment, test, or measure consistently yields the same result under similar conditions)
• Validity (the extend to which a concept, measurement, or conclusion is well founded and corresponds to real applications), in particular– Internal validity (is the relationship causal)– External validity (generalization of results)– Statistical validity (statistical relevance of conclusions)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 70
A (bad) example
Assume a (real) museum with paintings & music playing (indep. variable).• Week 1: play classical music• Week 2: play rock musicMeasure average time spent in museum per visitor (dependent variable)
Result (e.g.): • In week 1, people spend on average 1.5 hours in the museum• In week 2, they spend on average 5 hours(Wrong) conclusion: play rock music because it attracts more people
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 71
How about:• Internal & external validity? E.g.: Comparable sample sizes?
Comparable subject characteristics (e.g., age)? Comparable conditions (e.g., tour busses with fixed schedule)?
• Confounding factors? E.g.: What if it rained in week 2?• Statistical validity? E.g.: Do the sample groups have the same size?
Are the results statistically significant?
Analyzing measured data: Statistical validity
What to report? E.g.:• Averages (also means, variance, …)• Distributions• Significance analysis resultsAgain, you are allowed to make compromises (if you can explain and justify them)
Also: what conclusions can you draw from the data?E.g.: Does spending more time in a museum with different music really suggest people spent time with the paintings (or did they just dance in the cafeteria?)
Note: correlation is not the same as causationAnd again: don’t just gather data, but meaningful data
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 72
Study design (with respect to the research question)
• Subjects; who, how many, …– How much do you need for statistical relevance?– How many samples per subject?– You are allowed to make compromises; but explain them
• Within/between subject design– Advantages between subject design:
less subjects, enables qualitative comparison– Advantages within subject design:
shorter test duration, no carry over effects
• Order & data/condition mapping, ...– Counterbalanced, Latin-square design, ...
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 73
Another example: Immersion in VR
Use multimodality to improve immersion in VR• Is it specific? (What is immersion anyway?)• Is it measurable? (and how?)
Hint: if you can’t measure it, maybe there is a correlation between a measurable unit? E.g., some research suggests that a higher perception of presence has a positive impact on enjoyment.
Example: Presence in VR
See survey paper by Schuemie et al. (2001), Research on Presence in Virtual Reality: A Survey. Cyber Psychology & Behavior, Vol. 4(2).
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 74
Schuemie et al. (2001), Research on Presence in Virtual Reality: A Survey. Cyber Psychology & Behavior, Vol. 4(2).
1. Introduction2. On the nature of presence3. Results of presence4. Measuring presence5. Causes of presence6. Discussion
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 75
2. On the nature of presence Various definitions exist, e.g.,• With respect to transportation, i.e., being “present” in the
virtual world• Presence (being in a computer generated world) versus
telepresence (being at a real remote location)• Personal (I am there) vs. social (other beings also exist there)
vs. environmental presence (environment reacts to me)• Subjective (likelihood of a person judging himself as being
physically there) vs. objective presence (likelihood of successfully completing a task)
Also, other distinctions, e.g.:• Immersion as an objective description of aspects of the
system (e.g., FOV) vs. presence as a subjective phenomenon (the sensation of being there); Slater & Wilbur (1997)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 76
For your project: Specify concretely what you mean
3. Results of presenceTheories and empirical studies exist with respect to:• Subjective sensation of presence• Task performance• Responses and emotions• Simulator sicknessNot all of them are conclusive though.
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 77
For your project: Specify what concrete subdomain
you are addressing
4. Measuring presenceVarious approaches have been used in the literature:
• Subjective measures: questionnaires(note: various examples exist; see paper)
• Other subjective measures, e.g., continuous measure, presence counter, focus group exploration
• Objective measures: behavioral (e.g., reflex responses)• Objective measures: physiological (e.g., heart rate, skin
conductance, skin temperature)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 78
For your project: Find suitable, reliable, feasible
means that fit your goal
5. Causes of presenceResearch devoted to finding factors that contribute to presence in VR.
Different categorizations exist, e.g., by Steuer (1992)
• Vividness (the ability of a technology to produce a sensorially rich mediated environment)
– Breath: number of sensory sensations
– Depth: resolution within each perceptual channel
• Interactivity (the degree to which the user of a medium can influence the form or content of the VR)
• User characteristics (individual differences in users)
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 79
2/6/19 80
Research problemThe issue or problem within a broad topic area
Research purposeA statement of the intent or objective of the study
HypothesisA prediction or best guess of the relation that exists
among the variables being investigated
Research questionIn quantitative research it is an interrogative sentence
that asks a question about the relation between two or more variables. In qualitative research, it is an
interrogative sentence that asks a question about some process, issue, or phenomenon to be explored.
Research topicThe broad subject matter area to be investigated Relate this to user engagement
• What is it (formal definition)?• What specifies it, or what
contributes to it?• How can we measure that?
For your project:• Be clear what you are
focusing on • Describe it formally• Pick the right sub-questions,
tasks, and appropriate means
That will make it easier:• To do it successful• To relate your (very specific
and focused) outcome to the “bigger picture” / major goal
For your project
2/6/19 81
What (general) problem are you approaching?What concrete sub-problem will you address?How are you going to verify your ideas/solutions?
Three slides (strict; plus one intro & one closing).
1. Problem (general goal and concrete sub problem)
2. Solution (your idea to address the sub problem and/or the research question you want to answer, related hypothesis, and objectives)
3. Methodology (your planned approach to verify your results or proof that your solution works)
Time limit is 5 minutes (& will be strictly enforced).
Recommendation for brainstorming: Try answering these questions
See also info on the website!
Project pitch (Feb 25):
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 82
“Soft” goals (for project):• Self-dependent work, e.g.
– Coming up with own ideas– Being able to specify them into research questions– Making realistic time planning and scheduling
• Teamwork
Learning goals / objectives of the course (project):
Projects: after successful completion of this part of the course ...• Students can apply basic techniques of multisensory interfaces to
create a multimodal interaction, e.g., for control of a game (demonstrated by the related project work).
• Students know the advantages and disadvantages of using different media types and modalities depending on different contexts, conditions, and applications (demonstrated by the related project work).
• Ideally/normally, you will get a team grade– Exceptions apply for particularly motivated or lazy students
– Keep a log book tracking your hours and activities!– You only have to report your total hours and major
contributions/activities in the end (see report template)– Log book is needed if above exceptions need to be applied
– Note: You won’t get punished for other people’s failures(but you might get punished for badly dealing with them)
– Better grades than the group grade are possible (e.g., for cum laude students), but YOU have to convince me why you deserve it
You find more info on the grading in the document about the project report (see website). Read it now!
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 83
Some comments on grading
Lecture Project Recommended actions
Week 1Introduction
Papers 1, 5
Week 2 Lecture 1 (vision)Lecture 2 (sound, …) Deadline for forming groups
Papers 2, 3, 4 & forming project groups
Week 3 Lecture 3 (VR)Lecture 4 (other)
Papers 6, 7, 8 & work on project ideas, prepare pitches, plan
Week 4 Project pitchesGuest lecture
Deadline pitch slides: Feb 25, 9:00Deadline proj. plan: Feb 27, 23:59
Pitch ideas & make project plan
Week 5 Lecture 5 (AR)Lecture 6 (AR)
AR papers &setup and implementation
Week 6 Project presentationLecture 7 (AR)
Deadline slides: Mar 11, 9:00 AR papers &setup and implementation
Week 7 Lecture 8 (AR) Experiments
Week 8 Analysis & report / paper writing
Week 9Project presentation(final) & meetings
Deadline proj. report: Apr 1, 23:59
Examweek
Exam
1. Read info on the website (all documents, including the slides that I skipped today; the ones under “HELPFUL LITERATURE” are optional, but might come in handy later)
2. Form groups (via spreadsheets); also specify there if you want to borrow our hardware
3. Brainstorm ideas while following the lectures & reading papers
4. Start preparing project pitches (for Feb 25)Note that they don’t have to be perfect and can contain open questions to ask the audience!
5. Start working on project plans (due Feb 27)You find a template and further info on the website.
And most importantly: have fun and enjoy the course!
2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 85
Summary of next steps
If you have questions, email me ([email protected]).
Top Related