Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training
description
Transcript of Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training
![Page 1: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/1.jpg)
Paul Varcholik, Joseph J. LaViola Jr., Denise NicholsonInstitute for Simulation and TrainingUniversity of Central Florida
[email protected], [email protected], [email protected]
TACTUS: A Hardware and Software Framework for Research in Multi-Touch Interaction
![Page 2: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/2.jpg)
Multi-Touch: Introduction
![Page 3: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/3.jpg)
Framework Motivation Primary multi-touch components
1. Physical interaction surface2. Software system for collecting and
interpreting points of contact
Present a barrier to entry for researchers focused on higher-level interface issues or application development
![Page 4: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/4.jpg)
Multi-Touch Origins (pre-2000) Keyboards Touchscreens Pen-based computing 1984 – Bob Boie (Bell Labs) perhaps the first
multi-touch screen 1985 – Bill Buxton, Multi-Touch Tablet 1990 – Sensor Frame, Paul McAvinney (CMU)
Optical Sensor 3 fingers (some trouble with ambiguous finger positions)
1998 – Fingerworks
![Page 5: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/5.jpg)
Multi-Touch Origins (2000 to present) 2001 – Paul Deitz – Mitsubishi DiamondTouch 2003 – Jazz Mutant 2004 – Andy Wilson – TouchLight Oct. 2005 – Jeff Han – FTIR, UIST Paper Feb. 2006 – Jeff Han – TED Talk 2006 – Jeff Han – Perceptive Pixel (CNN Magic Wall) Spring 2007 – Microsoft Surface Summer 2007 – Apple iPhone 2008 – Open-Source Community 2009 – Windows 7
![Page 6: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/6.jpg)
Framework RequirementsEssential Component DescriptionMulti-Touch Surface Commercial-off-the-shelf hardware, requiring
near zero pressure to detect an interaction point
Software Hit-Testing Ability to determine the presence and location of each point of surface contact; supporting at least four concurrent users
Software Point Tracking Identifying a continuous point of contact and reporting its velocity and duration
![Page 7: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/7.jpg)
Framework RequirementsSecondary Component Description
Application Programming Interface (API)
A software system upon which multiple multi-touch applications can be developed
Multi-Platform Support The ability to access multi-touch interaction data from different computing environments (e.g. languages, OS’s, etc.)
Reconfiguration Modifying the software system without recompilation
Software Service Allowing multiple applications to access multi-touch interaction data simultaneously, including over a computer network
Presentation-Layer Independence
Isolating the multi-touch interaction data from the system to graphically present such data, allowing any GUI to be employed when developing multi-touch applications
Mouse Emulation Support for controlling traditional ‘Window, Icon, Menu, Pointing Device’ interaction through a multi-touch surface
Tangible Interfaces The ability to detect and interact with physical devices placed on or near the multi-touch surface
Customizable Gesture System Support for training arbitrary multi-touch gestures and mapping them to software events
Shape Detection e.g. Finger, blob, hand
![Page 8: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/8.jpg)
Multi-Touch Hardware Frustrated Total Internal Reflection (FTIR)
1. Optical waveguide2. Supporting structure (e.g. cabinet)3. IR sensing camera4. Projector and diffuser5. IR emission source6. Computer
![Page 9: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/9.jpg)
Multi-Touch Efforts: Hardware
![Page 10: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/10.jpg)
Software Framework Image processing Presentation layer Multi-platform communication Pen/writing style interaction (Ink) Gesture recognition
![Page 11: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/11.jpg)
Image Processing Pipeline
Webcam RotateFlip Background Filter
Threshold
Scaling
Blob Detection GrayScale
Grayscale Image 8bpp
Binary Image 1bpp
Calibration Point Tracking
ProcessedFrame
Camera Frame24bpp
![Page 12: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/12.jpg)
Image Processing
Raw Camera Frame
Background Filtered Threshold Processed
Points
![Page 13: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/13.jpg)
Multi-Touch Efforts: Software
![Page 14: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/14.jpg)
Mouse EmulatorFunction Gesture DescriptionLeft Click Quick tap on the surface with one finger.
Left Click (alternate) While holding down a finger, tap another finger to the left side of the first.
Drag Perform a Left Click (alternate) but do not release the left side press. Drag both fingers to the destination and release.
Right Click While holding down a finger, tap another finger to the right side of the first.
Double Click Tap two fingers at the same time.
Mouse Wheel Scroll While holding down a finger, drag another finger vertically and to the right side of the first. Dragging up scrolls the mouse wheel up and vice versa.
Alt-Tab While not a mouse command, the Alt-Tab command is a useful Windows feature that switches between applications. To perform an Alt-Tab, hold down a finger and drag another finger horizontally above the first. Dragging to the left moves backward through the list of active applications and dragging to the right moves forward.
![Page 15: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/15.jpg)
Multi-Touch Efforts: Video Montage
![Page 16: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/16.jpg)
Multi-Touch Efforts: Additional Videos Demo Explorer Improved Calibration Multi-Touch Starcraft Mouse Emulator Ink Demonstration Particle System Demonstration SurfaceSimon SurfaceCommand
![Page 17: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/17.jpg)
Conclusions The Framework:
In development since Fall 2007Open-source availability since Spring 2008Now in its fourth major releaseAnswers most of the specified requirementsSuccessfully employed by UCF and colleagues
from Brown UniversityDownloaded extensively by the open-source
communityProvides a robust, extensible platform for
research in multi-touch interaction
![Page 18: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/18.jpg)
Questions?
Paul Varcholik, Joseph J. LaViola Jr., Denise NicholsonInstitute for Simulation and TrainingUniversity of Central Florida
[email protected], [email protected], [email protected]
TACTUS: A Hardware and Software Framework for Research in Multi-Touch Interaction
![Page 19: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/19.jpg)
Resources: Commercial Platforms Microsoft Surface Mitsubishi DiamondTouch Apple iPhone & iPod Touch Perceptive Pixel N-trig MultiTouch OY (Finland) Jazz Mutant HP TouchSmart Dell Latitude XT2 TacTable
![Page 20: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/20.jpg)
Resources: Open-Source Platforms TouchLib (NUI Group) tBeta (NUI Group) reacTIVision Bespoke Multi-Touch Framework Sparsh UI (Iowa State) BBTouch Touche`
![Page 21: Paul Varcholik, Joseph J. LaViola Jr., Denise Nicholson Institute for Simulation and Training](https://reader036.fdocuments.us/reader036/viewer/2022062520/56816597550346895dd86e1e/html5/thumbnails/21.jpg)
Resources: Online NUI Group TouchKit by NOR_/D Multi-Touch Systems that I Have Known
and Loved – Bill Buxton, Microsoft Research
Jeff Han’s TED Talk Bespoke Software