OPTIMIZATION OF
GESTURE RECOGNITION
IN ITERACTIVE DIGITAL
SPACE
Mahdi Babaei1131600043
Research scholar – TM R&D Project
MASTER OF SCIENCE IN CREATIVE MULTIMEDIA
Supervisor: Assoc. Prof.Dr.Wong Chee Onn
Co-supervisor: Dr.Lim Yan Peng (Forest)
Work Completion Seminar
INTRODUCTION
OVERVIEW OF PRESENTATIONIntroductionResearch ObjectivesResearch QuestionsLiterature reviewMethodologiesSolution proposalResults and AnalysisConclusion
INTRODUCTION With growth in number of innovation in
new devices on 1990th HCI came to daily life.
From simple traditional GUI
To
Inconvenien
t &
unnatural
3D Virtual
SpaceHuman
GestureDigital
content in
curve
displays
RESEARCH QUESTION “How to optimize the best combination of gesture recognition methods in order to have an efficient system (from user view) while improving the quality of interaction and human factors, in a digital space?”
OBJECTIVES To study on gesture recognition methods and
design a taxonomy to categorize them. To study on Digital space, content and
projection and design a space based on the knowledge.
To study on virtual reality based interaction and environment, pick factors and design a suitable interaction based on that.
To test interactive gesture recognition prototype in virtual space with digital contents.
To solve the disadvantages of proposed combination method of gesture recognition in virtual space with digital content.
LITERATURE REVIEW
HUMAN GESTURES AND TYPES Gestures are expressive, meaningful
body motions involving physical movements of the fingers, hands, arms, head, face, or body.
Types based on(Billinghurst and Buxton 2011):
Gestures in everyday World.Gestures only for interfaces.
There are many taxonomies in this area like: (Kammer, Keck et al. 2010) (Karam 2005)
TAXONOMY OR CATEGORY ? We needed to design a new taxonomy of
gestures because:We need to go deep down in order to find
the exactly suitable gesture category that would match the criteria and category can cover basics only.
Taxonomy provides vocabulary and Tree-Node system and flexibility of adding new vocabularies. On the other hand category can offer a group of nodes with strict hierarchy.
QUALITATIVE HUMAN FACTORS OF GESTURE RECOGNITION
Based on (Barclay, Wei et al. 2011) :Fatigue (Barclay, Wei et al. 2011)
Intuitiveness (Nielsen, Störring et al. 2004)
UsabilityLearnability (Valkov, Steinicke et al. 2010)
Easiness to navigate and orientate (Valkov, Steinicke et al. 2010)
Naturalness and memorability (Lenman,
Bretzner et al. 2002).Ergonomic
QUANTITATIVE HUMAN FACTORS OF GESTURE RECOGNITION
Gesture time and duration(speed) (Barclay, Wei et al. 2011
Accuracy, Precision and error rate User Cooperation
GESTURE RECOGNITION DEVICES
Sensors
Mix
Vision Base
d Methods
and cameras
INTERACTION TYPES 2D Interaction
Two-Dimensional interaction in 3D world.
Three-Dimensional interaction
3D INTERACTION REQUIREMENTS 3D Interaction needs 3D space where
user would be able to have movements.
It needs 3D contents (output) to let user manipulate with them.
It needs 3D hardware tools (input) to detect depth value
HUMAN FACTORS OF INTERACTION
based on (Liu and Shrum 2002,Benyon 2010):Control desire (Burger and Cooper 1979, Liu and Shrum 2002)
AccessibilityComputer-mediated communication
apprehension (CMCA) (Liu and Shrum 2002)(The level of expertise in using computers)
Usability: Based on (Standardization 1998, van Kuijk 2012
): Effectiveness Efficiency Satisfaction
Acceptability
INTERACTIVITY OF GESTURES IN DIGITAL SPACES Based on (Hale and Stanney 2002) actions can
be:
Navigating through space.
Specifying item of interest.
Manipulating objects in the environment.
Changing object values.
Controlling virtual objects.
Issuing task-specific commands.
VIRTUAL REALITY ENVIRONMENTS AND DEVICES
Head-Mounted Displays
Cave Virtual Environment
Virtual re
ality workbench
HUMAN FACTORS OF VIRTUAL ENVIRONMENTS
Based on (Wilson and D’Cruz 2006) :
Influence of interaction on both sides.
User’s characteristics.
User’s needs.
METHODOLOGY AND PROPOSED SOLUTION
DESIGNED TAXONOMY
Virtual
Physical(epistemic)
Two-degree freedom for 2D interactionMultiple DOF for 2D interactionMultiple DOF for 3D interaction
Gestures with tangible objects for 3D interaction
Gestures for Real-World physical object interaction
Paralinguistic
Linguistic
Act
Symbol
Deictic Mimetic Gesticulations Metaphoric Affect displays Beat
Referential
Modelizing
Descriptive
Suggestive
Prompting
Emphatic
Side Effects of expressive behaviours
Mix-Communication Symbolic-Interactive
Human Gestures
Symbolic (based on One-Way Communication)
(Communicative or Semiotic)
Interactive (based on Two-Way communication) or
Manipulative Communication
Emblems/Illustrators
Iconic
Regulators
GESTURE DESIGN
Electromagneti
cAcoustic Optical Mechanical
Advantage - - Fast upload rate -
Disadvantag
e
High inference with
magnetic field
Low rate target
positioning
Sight can obscured or
interfered
Limits user’s range
of motion
3D Model BasedAppearance-based
Volumetric Skeletal
Speed
Low
(complexity
of calculation)
High
( only key parameters
are analyzed)
Medium
(depends on algorithm)
Accuracy High High Medium
Processing
timeHigh Low Low
Points complicated 3D surfaces Skeleton Joints extraction Shape extraction
SYSTEM DESIGN
Three-Dimensional
Virtual Environment
Gesture Recognition
Interaction
Two-Dimensional interaction in 3D
world
Two-Dimensional
3D- Hardware
3D- Space, 3D- Contents
System Control
Navigation
Selection
Object Manipulation Physical Movement
Manual viewport Manipulation
Steering
Target-based Travel Route planning
FACTOR INTERSECTION FINDING
Accessibility
Computer-mediated communication apprehension
Usability
• Effectiveness• Efficiency• Satisfaction
AcceptabilityInfluence on participantsParticipant’s InfluenceUser Characteristics and needs
Fatigue
Intuitiveness
Usability• Learnability• Easiness to
navigate and orientate
• Naturalness and memorability
• Ergonomic
Speed
Accuracy
User Cooperation
Virtual Environment
Gesture Recognition
Interaction
Control Desire
PROPOSED FRAMEWORK
Gesture recognition factors
Quantitative Factors Qualitative Factors
User expertise in using computer systems
Usability
FatigueIntuitiveness
User cooperation
Speed
Accuracy
Acceptability
Learnability
Easiness to navigate and orientate
Naturalness and memorability
Ergonomic
Control Desire
Accessibility
Satisfaction
Efficiency
Effectiveness
CHOOSING KINECTDevice Popularit
yMotor Driver SDK Image Quality Size Weight Power
Microsoft Kinect
High Has HQ HQ Medium 12"x 3" x 2.5"
3.0 lb Ac + DC
ASUS Xtion /
PrimeSense Carmine
Low
No LQ LQ HQ 7" x 2" x 1.5"
0.5 lb DC-USB
We choose to optimize Microsoft because: It has higher driver quality. It has software development kit.Popularity means easier access to research
resources.
KINECT WEAKNESS Can not track user’s eye and head
movements and rotations
PROPOSED SOLUTION A combination of Microsoft Kinect as
skeletal detection device and an acceleration or gyroscope data.
PROPOSED VE DESIGN
Normal V
iew
Maximize
d View
Cave Design
Implemented C
ave
PROPOSED GESTURE RECOGNITION STRUCTURE
Head and eye
Gestures
BodyGestures Kinect
Camera
AccelerometerDigital Receiver
Analogue Receiver
Analogue to Digital convertor
Antennas
ROTATION AND MOVEMENTS
Yaw
Pitch
Roll
Top/Left/ Bottom/ Right
PROPOSED GESTURE
Knee Heigh
t
Time
PROPOSED HARDWARE
RESULTS AND ANALYSIS
MEASUREMENT STATES Before(Using Microsoft Kinect Only) After(Using proposed combination)
FACTORS Quantitative based on:
Logical optimization.Optimization measurement based on
results.
QualitativeQuestionnaire
QUANTITATIVE MEASUREMENTS Speed
16.6% improvement in coverage angle
Min Max
-100
-60
-20
20
60
100
-60
60
-90
90
Before After
Degre
e
SPEED
Speed0
100
200
300
400
500
527.64
315.32
418.37
AHRS Kinect Proposed method
Rot
atio
n sp
eed
in o
ne
seco
nd(D
egre
e P
er S
econ
d)
Percentage0
102030405060708090
10099.38
59.39
78.80
AHRS Kinect Proposed method
Per
cent
age
of s
ucce
ssfu
l re
cogn
ized
ges
ture
103.05(d/s) improvement in
mean value average of
rotation speed (Degree Per
Second)
19.41 % improvement in mean value of successfully
recognized gestures in a
second
ACCURACY
Error rate percentage in one second0
102030405060708090
100
0.62
40.61
21.20
AHRS KinectProposed Combination
Err
or r
ate
Per
cent
age
19.41 % Reduction in mean value average of error rate.
QUALITATIVE FACTORS Questionnaire:
100 participant. 104 question (each factor 8
question).Same participant completed the
same questionnaire.Two steps: Before and AfterLikert 7 ScaleMean Value test
RESULTS TEST
Reliability StatisticsCronbach's
Alpha N of Items
0.710 104
Reliability StatisticsCronbach's
Alpha N of Items0.752 104
Reliability Test
BEFORE
AFTER
FACTOR 1- EVALUATION OF USER EXPERTISE IN USING COMPUTERS
Mean Average Standard Deviation Average
01234567
4.93
1.76
Overal User Expertiese
Like
rt 7
Sca
le
FACTOR 2 – USER’S FATIGUE
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.18
1.50
2.66
1.12
Before After
Like
rt 7
Sca
le
7.42 % Reduction in mean value average
FACTOR 3 – USER’S ERGONOMIC AND ANXIETY
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.73
1.63
3.54
1.73
Before After
Like
rt 7
Sca
le
15.05 % Reduction in mean value average
FACTOR 4 – LEARNABILITY
21.14% Improvement in Average of Mean value.
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.27
1.44
4.75
1.94
Before After
Like
rt 7
Sca
le
FACTOR 5 – ACCEPTABILITY
9.5% Improvement in average of mean value.
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
2.57
1.31
3.24
1.60
Before After
Like
rt 7
Sca
le
FACTOR 6 – EASINESS TO NAVIGATE AND ORIENTATE
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.67
1.54
5.21
1.62
Before After
22 % Increase in average of mean value
Like
rt 7
Sca
le
FACTOR 7 – USER CONTROL DESIRE OVER ENVIRONMENT
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.12
1.47
4.32
1.32
Before After
17.14 % Increase in average of mean value
Like
rt 7
Sca
le
FACTOR 8 – NATURALNESS AND MEMORABILITY
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.02
1.13
4.10
1.25
Before After
15.42 % Increase in average of mean value
Like
rt 7
Sca
le
FACTOR 9 – ACCESSIBILITY
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
2.60
1.13
3.14
1.14
Before After
7.71 % Increase in average of mean value
Like
rt 7
Sca
le
FACTOR 10 – SATISFACTION
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.98
1.51
4.76
1.81
Before After
11.42 % Increase in average of mean value
Like
rt 7
Sca
le
FACTOR 11 – EFFICIENCY
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.50
1.57
5.42
1.69
Before After
27.42 % Increase in average of mean value
Like
rt 7
Sca
le
FACTOR 12 – EFFECTIVENESS
27.85% improvement in average of mean value
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
3.23
1.10
5.18
1.44
Before After
Like
rt 7
Sca
le
FACTOR 13 – INTUITIVENESS
10.28% improvement in average of mean value
Mean Average Standard Deviation Average0
1
2
3
4
5
6
7
4.62
1.60
5.34
1.25
Before After
Like
rt 7
Sca
le
USABILITY
Usability
Sub-FactorMean Value Standard
DeviationBefore After Before After
Learnability 3.27 4.75 1.44 1.94
Easiness to navigate and orientate
3.67 5.21 1.54 1.62
Naturalness and memorability
3.02 4.10 1.13 1.25
Average 3.32 4.68 1.37 1.6
Before After01234567
3.32
1.37
4.68
1.6
Mean Value Standard Deviation
19.42% improvement in average of mean value and more usable gesture recognition.
Like
rt 7
Sca
le
USER COOPERATION
User Cooperation
Sub-FactorMean Value Standard
DeviationBefore After Before After
User Control Desire over environment
- 4.92 - 1.76
Accessibility 2.57 3.24 1.31 1.6Satisfaction 3.98 4.76 1.51 1.81
Efficiency 3.5 5.42 1.57 1.69Effectiveness 3.23 5.18 1.10 1.44
Average 3.32 4.704 1.37 1.66
19.77% improvement in average of mean
Before After0
1
2
3
4
5
6
7
3.32
1.3725
4.704
1.66
Mean Value Standard Deviation
Like
rt 7
Sca
le
ALL FACTORS
Factor Before AfterAbsolute value of change
Percentage
Effectiveness 3.23 5.18 1.95 27.86
Efficiency 3.5 5.42 1.92 27.43
Easiness to navigate and orientate
3.67 5.21 1.54 22.00
Learnability 3.27 4.75 1.48 21.14
User Control Desire over environment
3.12 4.32 1.2 17.14
Naturalness and memorability 3.02 4.1 1.08 15.43
Satisfaction 3.98 4.76 0.78 11.14
Intuitiveness 4.62 5.34 0.72 10.29
Acceptability 2.57 3.24 0.67 9.57
Accessibility 2.59 3.14 0.55 7.86
Fatigue 3.18 2.66 0.52 7.43
Ergonomic and anxiety 3.73 3.54 0.19 2.71
CONTRIBUTIONS Wireless interactive gesture recognizer
device
Gesture design contribution.
High speed in tracking.
High transmission speed.
Designed Taxonomy and Framework.
CONCLUSION
Qualitative factors general optimization percentage average: 14%
Quantitative factors general optimization percentage average: 19.91%
THANKYOU
Q & A
Top Related