2 December 2005
Fusion in Multimodal Interactive Systems: An HMM-Based Algorithm for User-Induced Adaptation
Bruno Dumas1, Beat Signer1 and Denis Lalanne2
1 WISE Research Lab, Vrije Universiteit Brussel, Belgium2 DIVA Research Group, University of Fribourg, Switzerland
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 2June 26, 2012
Outline• Fusion of multimodal input
• Multimodal interaction: from design to implementation• Algorithms for fusion of multimodal input
• A Hidden Markov Model-based fusion algorithm• Instantiation of the algorithm
• Evaluation of the algorithm• Qualitative test• Quantitative evaluation• Performance assessment
• Conclusion
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 3June 26, 2012
Fusion of Multimodal Input
Challenges in multimodal interaction engineering: continous and probabilistic inputs, real-time…
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 4June 26, 2012
Challenging/Ambiguous Fusion Cases
• “Put-that-there” Complementarity case
• “Play next track” Different meanings based on order and context
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 5June 26, 2012
CARE Properties at the Design Level
Describe how modalities can be combined Complementarity
All modalities are necessary
Assignment Absence of choice
Redundancy Same expressive power between modalities
Equivalence Any modality is sufficient
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 6June 26, 2012
HephaisTK Multimodal Framework
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 7June 26, 2012
Description of Multimodal Dialogues
• Syntactic-level description
• For more details: check SMUIML language
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 8June 26, 2012
Fusion Algorithms
• Frame-based algorithms & Unification-based algorithms• Symbolic approaches• Both are a de facto standard
• Symbolic-statistical algorithms
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 9June 26, 2012
Symbolic-Statistical Approaches
Symbolic data Interpretation of low-level data
Statistical processing Machine learning algorithms
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 10June 26, 2012
Our HMM-Based Fusion Algorithm
Why hidden Markov models? Very good modelling of time-related events Takes advantage of input data coming from
probabilistic input modalities On-the-fly adaptation of the model
- e.g. based on user feedback
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 11June 26, 2012
Put-That-There Example
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 12June 26, 2012
Put-That-There Example
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 13June 26, 2012
Put-That-There Example
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 14June 26, 2012
Put-That-There Example
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 15June 26, 2012
Put-That-There Example
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 16June 26, 2012
Put-That-There Example
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 17June 26, 2012
Instantiation of the Algorithm
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 18June 26, 2012
Evaluation of the Algorithm
Evaluation on three levels Qualitative “gut check” test Benchmarks on real-world examples Performance assessment
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 19June 26, 2012
Evaluation: Qualitative Test
• Three expert users were asked to assess the behaviour of the algorithm frames-based and HMM-based fusion 5 minutes per condition + interview
• Coherent behaviour between the algorithms
• No noticeable difference in responsiveness
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 20June 26, 2012
Evaluation: Benchmark
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 21June 26, 2012
Evaluation: Benchmark
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 22June 26, 2012
Benchmark Results (1)
• CARE properties: equivalence and redundancy tests
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 23June 26, 2012
Benchmark Results (2)
• Sequential and non-sequential complementarity tests
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 24June 26, 2012
Benchmark Results (3)
• “Play next track” example
Mea
ning
fra
mes
HM
M-b
ased
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 25June 26, 2012
Evaluation: Performance
• Per fusion algorithm: • 5 runs of 40 input pieces• 5 x 20 expected fusion results
• Frame-based: 18.2 ms• Standard deviation: 12.7 ms
• HMM-based: 16.6 ms• Standard deviation: 11.6 ms
Bruno Dumas – WISE resarch lab – Department of Computer Science - [email protected] 26June 26, 2012
Conclusion
• A new symbolic-statistical multimodal fusion algorithm based on Hidden Markov Models
• Integrated into the HephaisTK framework
• Superior recognition rates compared to existing algorithms
• Handling of ambiguous cases
• Processing of real-time user feedback
http://wise.vub.ac.be/bruno-dumas
Top Related