Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent...
Transcript of Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent...
![Page 1: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/1.jpg)
T. Metin Sezgin
Assoc. Prof. College of Engineering Koç University
http://iui.ku.edu.tr [email protected]
BYOYO 01/07/20
Psychology-Driven Design of Intelligent Interfaces
![Page 2: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/2.jpg)
Dr. Metin Sezgin, Assoc. Prof.
Intelligent User Interfaces Group
▪ Areas of expertise ▪ Intelligent User Interfaces ▪ Machine learning ▪ Multimodal interfaces
MIT (MS ‘01) MIT (PhD ‘06) postdoc visiting appointments 2010 -- …
▪ 20+ graduate students ▪ ~15 TL million sponsored projects
▪International ▪European Union ▪CHIST-ERA ▪DARPA
▪National ▪Research Council of Turkey ▪Ministry of Science, Industry & Tech.
▪Industrial ▪Türk Telekom ▪Koç Sistem
![Page 3: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/3.jpg)
History of Human Computer Interaction
2007
ENIAC
1946 1989
Mac Portable
1981
IBM PC iPhone
Cheaper: 13000 timesSmaller: 986530 timesFaster: 6077922 times
1017 Fold cost of flops/grams
![Page 4: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/4.jpg)
Attempts at intelligent interaction
Television Control by Hand Gestures William T. Freeman, Craig D. WeissmanMERL Report: TR94-24
![Page 5: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/5.jpg)
Attempts at intelligent interaction
Freeman ’94
Attempts at intelligent interaction have failed!
Solution: leverage natural human behavior
Unidentified Samsung User ’14
![Page 6: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/6.jpg)
The Problem Too little effort towards understanding interaction
![Page 7: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/7.jpg)
Strategy: Leverage natural human behavior
HCI
![Page 8: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/8.jpg)
Strategy: Leverage natural human behavior
HCI Machine learning
![Page 9: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/9.jpg)
Strategy: Leverage natural human behavior
HCI Machine learning
Psychology
![Page 10: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/10.jpg)
Strategy
▪ Understand the human ▪ Perception ▪ Behaviour
▪ Computational models of ▪ Human perception ▪ Human behavior (intent)
▪ Build novel interfaces (HW & SW) ▪ Natural ▪ Intelligent ▪ Multimodal
![Page 11: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/11.jpg)
Case #1
▪ Exercise ▪ Draw objects
▪ Observe human behavior
![Page 12: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/12.jpg)
Observe human behavior
![Page 13: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/13.jpg)
Case #1
▪ Exercise ▪ Draw objects
▪ Observe human behavior ▪ Practical use ▪ Sketch recognition ▪ Auto-completion of drawings
![Page 14: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/14.jpg)
Online Sketch Recognition
![Page 15: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/15.jpg)
Offline Sketch Recognition
Work Funded under the National Science Foundation Priority Areas Call
![Page 16: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/16.jpg)
Auto-completion
![Page 17: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/17.jpg)
T. M. Sezgin and R. Davis, Sketch Recognition in Interspersed Drawings Using Time-Based Graphical Models. Computers & Graphics Journal, Volume 32 , Issue 5, pp: 500-510 (2008). Ç. Tırkaz, B. Yanıkoğlu, T. M. Sezgin, Sketched Symbol Recognition with Auto Completion. Pattern Recognition, vol 45, issue 11, pp 3926-3937 (2012).
Auto-completion
![Page 18: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/18.jpg)
Auto-completion
▪ Drives multimedia retrieval UI ▪ iMotion European Commission ERA-NET Project
▪ U. Basel (Switzerland) ▪ U. Mons (Belgium)
Retrieval Engine
Grant: European Commission ERA-Net Program, CHIST-ERA Intelligent User Interfaces Call Intelligent Multimodal Augmented Video Motion Retrieval System
![Page 19: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/19.jpg)
Strategy: Leverage natural human behavior
![Page 20: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/20.jpg)
Case #2
▪ Exercise ▪ Object manipulation
![Page 21: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/21.jpg)
Object Manipulation
Virtual Interaction Task – Drag
![Page 22: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/22.jpg)
Virtual Interaction Task – Maximize
![Page 23: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/23.jpg)
Virtual Interaction Task – Minimize
![Page 24: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/24.jpg)
Virtual Interaction Task – Scroll
![Page 25: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/25.jpg)
Virtual Interaction Task – Free-Form Drawing
• Ππ
![Page 26: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/26.jpg)
Virtual Interaction Task: Your turn
![Page 27: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/27.jpg)
Case #2
▪ Exercise ▪ Object manipulation
▪ Observe human behavior
![Page 28: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/28.jpg)
Case #2
![Page 29: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/29.jpg)
Case #2
![Page 30: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/30.jpg)
Case #2
Ç. Çığ, T. M. Sezgin, Gaze-Based Prediction of Pen-Based Virtual Interaction Tasks. International Journal of Human-Computer Studies, (2014).
European Patent Application, T. M. Sezgin, Ç. Çığ, Gaze Based Prediction Device, PCT/TR2014/00189, European Patent Office, May 2014.
![Page 31: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/31.jpg)
Case #2
▪ Exercise ▪ Manipulate objects
▪ Observe human behavior ▪ Practical use ▪ Proactive UIs ▪ Intent recognition ▪ Fat finger problem
![Page 32: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/32.jpg)
Novel use of eye gaze How do I detect recognition errors?
![Page 33: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/33.jpg)
Novel use of eye gaze
▪ Immediate return to the misrecognition
![Page 34: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/34.jpg)
Novel use of eye gaze
▪ Immediate return to the misrecognition
▪ Double take at the misrecognition
![Page 35: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/35.jpg)
Novel use of eye gaze
▪ Immediate return to the misrecognition
▪ Double take at the misrecognition
![Page 36: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/36.jpg)
Research highlights
▪ Recognition technologies ▪ Perception-based ▪Machine learning
▪ Multimodal interaction ▪ Development ▪ Evaluation
![Page 37: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/37.jpg)
Recognizing Sketches
Grant: Funded under the National Science Foundation Priority Areas Call
![Page 38: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/38.jpg)
Learning a scale of messiness
vs.
![Page 39: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/39.jpg)
Recognition with few examples, scarce resources
CLUSTER A CLUSTER CCLUSTER B
▪ Modeling styles ▪ Active learning ▪ Zero shot learning ▪ Self learning
![Page 40: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/40.jpg)
Pen-based interfaces
▪ Design ▪ E-learning ▪ Animation ▪ Entertainment
![Page 41: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/41.jpg)
Multimodal Storyboarding Assistant
![Page 42: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/42.jpg)
Intelligent User Interfaces
smart stylus
rehabilitation of autism conditions
multimedia retrieval
gaze-based intent recognition
affective robotics HRI
SEARCH
![Page 43: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/43.jpg)
Grant: European Commission ERA-Net Program, CHIST-ERA Intelligent User Interfaces Call Joke and Empathy of a Robot/ECA: Towards Social and Affective Relations with a Robot
Affective interaction with robots
▪ Robots with a sense of humor ▪ JOKER – European Commission ERA-NET Project
▪ LIMSI/CNRS (France) ▪ Trinity College Dublin (Ireland) ▪ University of Mons (Belgium)
![Page 44: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/44.jpg)
Affective interaction with robots
Grant: European Commission ERA-Net Program, CHIST-ERA Intelligent User Interfaces Call Joke and Empathy of a Robot/ECA: Towards Social and Affective Relations with a Robot
![Page 45: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/45.jpg)
Affective interaction with robots
Grant: European Commission ERA-Net Program, CHIST-ERA Intelligent User Interfaces Call Joke and Empathy of a Robot/ECA: Towards Social and Affective Relations with a Robot
![Page 46: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/46.jpg)
Learning, vision, language
deep stroke segmentation
learning visual attributes
learning from few examples
active learning
explainable AI shape retrieval
![Page 47: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/47.jpg)
Looking forward
Medicine Social Sciences
Arts
![Page 48: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/48.jpg)
Alumni ProfilesDr. Yusuf Sahillioğlu, Visiting Researcher Assoc. Prof., Middle East Technical Univ.
Dr. Başak Alper, PostdocNASA - Jet Propulsion Laboratory
Neşe Alyüz Çivitci, PostdocIntel Labs, Intel Corporation
Senem Ezgi Emgin, PhD Student Apple
Zana Buçinca, MS Student Harvard University
Çağlar Tırkaz, PhD Student Amazon
Ayşe Küçükyılmaz, PhD Student Nottingham University (Asst. Prof.)
Kurmanbek Kaiyrbekov, MSc Student John Hopkins University
Cansu Şen, MSc Student University of Massachusetts Med. School
Tuğrulcan Elmas, Summer Researcher École Polytech. Fédérale de Lausanne
Arda İçmez, Summer Researcher Facebook
Mustafa Emre Acer, Summer Researcher Google
![Page 49: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/49.jpg)
Acknowledgements
▪ Postdocs ▪ Basak Alper ▪ Nese Alyuz ▪ Yusuf Sahillioglu
▪ PhD students ▪ Sinan Tumen ▪ Berker Turker ▪ Ayse Kucukyilmaz ▪ Caglar Tirkaz ▪ Cagla Cig ▪ Ezgi Emgin
▪ MS students ▪ Serike Cakmak ▪ Ozem Kalay ▪ Cansu Sen ▪ Erelcan Yanik ▪ Atakan Arasan ▪ Banucicek Gurcuoglu ▪ Kemal Tugrul
▪ Undergraduate students ▪ Anil Uluturk ▪ Furkan Bayraktar ▪ Ozan Okumusoglu ▪ 30+
▪ Collaborators ▪ Berrin Yanikoglu ▪ Engin Erzgin ▪ Yucel Yemez ▪ Cagatay Basdogan
▪ Sponsors ▪ DARPA ▪ The European Commission ▪ TÜBİTAK ▪ Türk Telekom ▪ Koç Sistem ▪ Ministry of Science
Industry & Technology
![Page 50: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/50.jpg)
Questions
![Page 51: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/51.jpg)
Questions
![Page 52: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/52.jpg)
References
Invention Disclosures Under review, O. Kalay., T. M. Sezgin, BBF # 2014.10.X Koç University, Research, Project Development and Technology Transfer Directorate
Gaze-Based Mode Inference for Pen-Based Interaction, Ç. Çığ, T. M. Sezgin, BBF # 2013.03.002Koç University, Research, Project Development and Technology Transfer Directorate
Auto-Completion in Sketch Recognition, T. M. Sezgin, B.Yanıkoğlu, Ç. Tırkaz, BBF # 2011.03.XKoç University, Research, Project Development and Technology Transfer Directorate
European Patent Application, T. M. Sezgin, Ç. Çığ, Gaze Based Prediction Device, PCT/TR2014/00189, European Patent Office, May 2014.
Publications Ç. Çığ, T. M. Sezgin, Gaze-Based Virtual Task Predictor. Proceedings of International Conference on Multimodal Interfaces, Workshop Eye Gaze in Intelligent Human Machine Interaction: Eye-Gaze and Multimodality, Accepted for publication (2014).
Ç. Çığ, T. M. Sezgin, Gaze-Based Prediction of Pen-Based Virtual Interaction Tasks. International Journal of Human-Computer Studies, Accepted for publication, (2014).
Ç. Tırkaz, B. Yanıkoğlu, T. M. Sezgin, Sketched Symbol Recognition with Auto Completion. Pattern Recognition, vol 45, issue 11, pp 3926-3937 (2012).
![Page 53: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/53.jpg)
History of Human Computer Interaction
2007
ENIAC
1946 1989
Mac Portable
1981
IBM PC iPhone
![Page 54: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/54.jpg)
![Page 55: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/55.jpg)
![Page 56: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/56.jpg)
Wizard of Oz Method
56
![Page 57: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/57.jpg)
The confession
Television Control by Hand Gestures William T. Freeman, Craig D. WeissmanMERL Report: TR94-24
![Page 58: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/58.jpg)
The Problem Too little effort towards understanding interaction
![Page 59: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/59.jpg)
Case study 3
![Page 60: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/60.jpg)
Case study 3
![Page 61: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/61.jpg)
Case study 3
![Page 62: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/62.jpg)
Case study 3
![Page 63: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/63.jpg)
Case study 3
![Page 64: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/64.jpg)
Case study 3
![Page 65: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/65.jpg)
Natural & Intelligent User Interfaces
▪ Understanding the user ▪ Natural modalities ▪ Collecting realistic data (observe the user in her space) ▪ Meet the user needs
▪ Real-time, seamless interaction ▪ Predictive interfaces
▪ Understanding machine learning ▪ Adaptation to the user ▪ Labeling large data sets (active learning) ▪ Getting better accuracies
▪ Classifier combination ▪ Feature selection
▪ Co-training, active learning ▪ Co-reference resolution
Important theme…
![Page 66: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/66.jpg)
Electrical Engineers Draw Sketches
![Page 67: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/67.jpg)
Electrical Engineers Draw Sketches
![Page 68: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/68.jpg)
Mechanical Engineers
![Page 69: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/69.jpg)
Software Engineers
![Page 70: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/70.jpg)
Sketch recognition
NPN Transistor
▪ Sketches are: ▪ Informal ▪ Messy ▪ Highly variable
▪ Focus: ▪ Iconic objects ▪ Compositional and hierarchical ▪ Online sketching (incremental)
▪ Our goal is to find: ▪ The correct segmentation ▪ The correct class
![Page 71: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/71.jpg)
After each stroke is drawn…
Sketch recognition at work
![Page 72: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/72.jpg)
Sketched Symbol Recognition with Auto-Completion
… a list of top matches is produced.
Predictive interfaces, Feature selection, Classifier combination, Co-training
![Page 73: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/73.jpg)
Input by
stylus
Recognition
Sketch recognition at work
Seamless integration, Natural modalities
![Page 74: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/74.jpg)
Machine learning technology & IUIs Designing interfaces and data collection
Realistic data, Seamless integration Wizard-of-Oz, Real-time interaction
![Page 75: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/75.jpg)
Machine learning technology & IUIs Data labeling
![Page 76: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/76.jpg)
Machine learning technology & IUIs Data labeling
Active learning
![Page 77: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/77.jpg)
Machine learning technology & IUIs User styles
CLUSTER A CLUSTER B
![Page 78: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/78.jpg)
Adaptation to user styles
Machine learning technology & IUIs Data labeling, User styles
![Page 79: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/79.jpg)
Led TV
Tobii X120Tablet
Multimodal Input and IUIs
Multimodality matters
![Page 80: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/80.jpg)
Multimodal Input and IUIs
![Page 81: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/81.jpg)
MIRA – Multi-Modal to Road Design Assistant
Natural modalities, Seamless integration Coreference Resolution
![Page 82: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/82.jpg)
MISA – A Multi-Modal Approach to Storyboard Design
Speech Input
Animation Recognition
Play the Slides
Animation
Natural modalities, Seamless integration Coreference Resolution
![Page 83: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/83.jpg)
Affect Recognition
![Page 84: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/84.jpg)
Helping Children with Autism Spectrum Conditions
![Page 85: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/85.jpg)
Helping Children with Autism Spectrum Conditions
![Page 86: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/86.jpg)
▪ 1% of the population ▪ Emotion recognition ▪ Display of emotions
▪ Learning through games ▪ Rehabilitation at a young age ▪ Interactive learning ▪ Formative assessment
▪ Approach ▪ Affect recognition ▪ Artificial intelligence ▪ Intelligent ingerfaces
▪ FP7 ASC-Inclusion ▪ International team (9 partners: Cambridge U., TUM … ) ▪ Academic, clinical, commercial impact ▪ Invaluable for the disadvantaged minorities
14/15
ASC-Inclusion
![Page 87: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/87.jpg)
Affective interfaces – animation
![Page 88: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/88.jpg)
Affective interfaces – recognition
![Page 89: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/89.jpg)
Affective interfaces – multiple modalities
![Page 90: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/90.jpg)
Collaboration and Negotiation: humans vs. computers vs. robots
Know thy customer! Modalities matter!
![Page 91: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/91.jpg)
Goals
▪ Create IUI awareness ▪Not just machine learning ▪Million ways to do it wrong
▪ Showcase technology ▪Sketch recognition ▪Multimodal interfaces
▪ Eye-gaze ▪ Speech ▪ Sketching ▪ Affect ▪ Haptics
![Page 92: Psychology-Driven Design of Intelligent Interfaces...Psychology-Driven Design of Intelligent Interfaces. Dr. Metin Sezgin, Assoc. Prof. Intelligent User Interfaces Group](https://reader033.fdocuments.us/reader033/viewer/2022042803/5f4ed61c8fe1ad05c06883b7/html5/thumbnails/92.jpg)
Acknowledgements
▪ Postdocs ▪ Basak Alper ▪ Nese Alyuz ▪ Yusuf Sahillioglu
▪ PhD students ▪ Sinan Tumen ▪ Ayse Kucukyilmaz ▪ Caglar Tirkaz ▪ Cagla Cig ▪ Ezgi Emgin ▪ Emre Karaman ▪ Ferhat Cagan
▪ MS students ▪ Cansu Sen ▪ Burak Ozen ▪ Ozem Kalay ▪ Erelcan Yanik ▪ Atakan Arasan ▪ Banucicek Gurcuoglu ▪ Kemal Tugrul
▪ Undergraduate students ▪ Anil Uluturk ▪ Furkan Bayraktar ▪ Ozan Okumusoglu
▪ Collaborators ▪ Cagatay Basdogan ▪ Berrin Yanikoglu ▪ Engin Erzin ▪ Yucel Yemez
▪ Sponsors ▪ DARPA ▪ European Commission ▪ National Science Foundation ▪ Türk Telekom ▪ Koç Sistem ▪ Ministry of Science
Industry & Technology