Midas: A Declarative Multi-Touch Interaction Framework
15
A Declarative Multi-Touch Interaction Framework CHRISTOPHE SCHOLLIERS LODE HOSTE BEAT SIGNER WOLFGANG DE MEUTER VRIJE UNIVERSITEIT BRUSSEL #MidasFW MIDAS
-
Upload
beat-signer -
Category
Science
-
view
4.586 -
download
1
description
Presentation given at TEI 2011, 5th International Conference on Tangible, Embedded and Embodied Interaction, Madeira, Portugal, January 2011 ABSTRACT: Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple’s iPhone or Microsoft’s Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hardcoded procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.
Transcript of Midas: A Declarative Multi-Touch Interaction Framework
A
Declarative
Multi-Touch
Interaction
Framework
CHRISTOPHE SCHOLLIERS
LODE HOSTE
BEAT SIGNER
WOLFGANG DE MEUTER
VRIJE UNIVERSITEIT BRUSSEL
#MidasFW
MIDAS
MULTI-TOUCH IS EVERYWHERE!
HARDWARE CAPABILITIES VS
GESTURE RECOGNITION SOFTWARE
Fl ick
Drag
Rotate
Pinch Stretch
WHAT’S THE FUNDAMENTAL PROBLEM?
update 0 0,565000 0,335417
update 1 0,387500 0,450000
update 0 0,560000 0,345833
update 1 0,390000 0,462500
update 0 0,558750 0,352083
update 1 0,391250 0,479167
update 0 0,556250 0,358333
update 1 0,391250 0,489583
update 0 0,553750 0,366667
update 1 0,392500 0,502083
update 0 0,550000 0,377083
update 1 0,392500 0,516667
update 0 0,548750 0,385417
update 1 0,392500 0,531250
update 0 0,546250 0,395833
update 1 0,390000 0,543750
update 0 0,543750 0,408333
update 1 0,388750 0,556250
update 0 0,542500 0,422917
update 1 0,386250 0,566667
update 0 0,543750 0,429167
update 1 0,385000 0,577083
update 0 0,543750 0,441667
update 1 0,378750 0,595833
update 0 0,543750 0,450000
update 1 0,367500 0,633333
update 0 0,547500 0,475000
update 1 0,348750 0,660417
update 0 0,551250 0,504167
update 1 0,341250 0,666667
update 0 0,553750 0,514583
update 1 0,336250 0,675000
update 0 0,557500 0,527083
update 1 0,331250 0,679167
update 0 0,561250 0,541667
update 1 0,326250 0,683333
update 0 0,565000 0,552083
update 1 0,321250 0,685417
update 0 0,570000 0,572917
update 1 0,316250 0,685417
update 0 0,575000 0,583333
update 1 0,311250 0,685417
update 0 0,581250 0,595833
update 1 0,307500 0,683333
update 0 0,590000 0,612500
update 1 0,302500 0,681250
update 0 0,595000 0,618750
update 1 0,296250 0,679167
update 0 0,606250 0,633333
update 1 0,290000 0,675000
update 0 0,612500 0,639583
update 1 0,281250 0,664583
update 0 0,622500 0,654167
update 1 0,272500 0,656250
update 0 0,631250 0,660417
update 1 0,268750 0,647917
update 0 0,636250 0,662500
update 1 0,262500 0,639583
update 0 0,643750 0,664583
update 1 0,257500 0,629167
update 0 0,647500 0,664583
update 1 0,255000 0,627083
update 0 0,653750 0,666667
update 1 0,166250 0,343750
update 0 0,728750 0,306250
update 1 0,172500 0,331250
update 0 0,725000 0,300000
update 1 0,177500 0,318750
update 0 0,723750 0,295833
update 1 0,181250 0,314583
update 0 0,720000 0,291667
update 1 0,185000 0,310417
update 0 0,718750 0,289583
update 1 0,186250 0,304167
update 0 0,716250 0,287500
update 1 0,190000 0,302083
update 0 0,713750 0,281250
update 1 0,192500 0,300000
update 0 0,708750 0,277083
update 1 0,198750 0,293750
update 0 0,702500 0,270833
update 1 0,205000 0,289583
update 0 0,692500 0,266667
update 1 0,208750 0,289583
update 0 0,688750 0,264583
update 1 0,212500 0,287500
update 0 0,685000 0,264583
update 1 0,216250 0,285417
update 0 0,678750 0,256250
update 1 0,221250 0,283333
update 0 0,675000 0,252083
update 1 0,227500 0,283333
update 0 0,671250 0,250000
update 1 0,231250 0,281250
update 0 0,667500 0,250000
update 1 0,233750 0,281250
update 0 0,662500 0,247917
update 1 0,238750 0,279167
update 0 0,658750 0,247917
update 1 0,245000 0,279167
update 0 0,655000 0,245833
update 1 0,248750 0,277083
update 0 0,651250 0,245833
update 1 0,253750 0,277083
update 0 0,646250 0,250000
update 1 0,258750 0,275000
update 0 0,642500 0,250000
update 1 0,263750 0,272917
update 0 0,638750 0,254167
update 1 0,267500 0,270833
update 0 0,635000 0,256250
update 1 0,271250 0,270833
update 0 0,631250 0,258333
update 1 0,278750 0,270833
update 0 0,625000 0,262500
update 1 0,282500 0,270833
update 0 0,622500 0,262500
update 1 0,286250 0,270833
update 0 0,621250 0,264583
update 1 0,288750 0,268750
update 0 0,620000 0,266667
update 1 0,291250 0,268750
update 0 0,618750 0,268750
update 1 0,292500 0,268750
update 0 0,617500 0,268750
update 1 0,293750 0,268750
update 1 0,251250 0,620833
update 0 0,656250 0,666667
update 1 0,242500 0,604167
update 0 0,663750 0,662500
update 1 0,237500 0,591667
update 0 0,667500 0,658333
update 1 0,232500 0,583333
update 0 0,672500 0,656250
update 1 0,215000 0,558333
update 0 0,691250 0,637500
update 1 0,206250 0,543750
update 0 0,702500 0,612500
update 1 0,187500 0,510417
update 0 0,711250 0,589583
update 1 0,167500 0,485417
update 0 0,716250 0,568750
update 1 0,161250 0,477083
update 0 0,720000 0,556250
update 1 0,152500 0,466667
update 0 0,725000 0,539583
update 1 0,146250 0,460417
update 0 0,727500 0,527083
update 1 0,143750 0,456250
update 0 0,728750 0,510417
update 1 0,138750 0,447917
update 0 0,731250 0,502083
update 1 0,136250 0,445833
update 0 0,733750 0,485417
update 1 0,133750 0,441667
update 0 0,736250 0,472917
update 1 0,132500 0,439583
update 0 0,740000 0,458333
update 1 0,131250 0,435417
update 0 0,741250 0,447917
update 0 0,741250 0,439583
update 1 0,130000 0,431250
update 0 0,742500 0,422917
update 1 0,130000 0,429167
update 0 0,742500 0,416667
update 1 0,130000 0,427083
update 0 0,743750 0,402083
update 1 0,130000 0,425000
update 0 0,745000 0,395833
update 1 0,131250 0,422917
update 0 0,745000 0,387500
update 1 0,132500 0,418750
update 0 0,745000 0,383333
update 1 0,133750 0,416667
update 0 0,743750 0,377083
update 1 0,135000 0,412500
update 0 0,743750 0,360417
update 1 0,136250 0,408333
update 0 0,741250 0,354167
update 1 0,141250 0,400000
update 0 0,738750 0,337500
update 1 0,145000 0,389583
update 0 0,738750 0,333333
update 1 0,147500 0,383333
update 0 0,735000 0,325000
update 1 0,151250 0,372917
update 0 0,731250 0,316667
update 1 0,157500 0,360417
update 0 0,728750 0,308333
add 0 0,633750 0,293750
update 0 0,635000 0,293750
update 0 0,633750 0,293750
update 0 0,633750 0,295833
update 0 0,633750 0,297917
update 0 0,632500 0,300000
update 0 0,631250 0,300000
update 0 0,630000 0,300000
update 0 0,628750 0,300000
update 0 0,627500 0,300000
add 1 0,246250 0,289583
update 1 0,247500 0,289583
update 1 0,248750 0,291667
update 1 0,250000 0,291667
update 1 0,252500 0,291667
update 1 0,256250 0,291667
update 1 0,258750 0,291667
update 1 0,263750 0,291667
update 0 0,626250 0,300000
update 1 0,276250 0,291667
update 0 0,625000 0,300000
update 1 0,285000 0,295833
update 0 0,622500 0,300000
update 1 0,293750 0,300000
update 0 0,621250 0,302083
update 1 0,300000 0,304167
update 1 0,310000 0,310417
update 0 0,618750 0,302083
update 1 0,318750 0,316667
update 0 0,616250 0,302083
update 1 0,325000 0,320833
update 0 0,613750 0,302083
update 1 0,330000 0,322917
update 0 0,611250 0,302083
update 1 0,335000 0,325000
update 0 0,610000 0,302083
update 1 0,338750 0,329167
update 0 0,607500 0,304167
update 1 0,341250 0,333333
update 0 0,605000 0,304167
update 1 0,345000 0,335417
update 0 0,602500 0,304167
update 1 0,347500 0,337500
update 0 0,600000 0,304167
update 1 0,351250 0,341667
update 0 0,596250 0,304167
update 1 0,355000 0,347917
update 0 0,593750 0,306250
update 1 0,358750 0,352083
update 0 0,590000 0,308333
update 1 0,361250 0,358333
update 0 0,586250 0,312500
update 1 0,365000 0,368750
update 0 0,583750 0,314583
update 1 0,368750 0,379167
update 0 0,580000 0,316667
update 1 0,372500 0,389583
update 0 0,577500 0,318750
update 1 0,378750 0,410417
update 0 0,571250 0,329167
update 1 0,382500 0,420833
update 0 0,568750 0,333333
update 1 0,383750 0,435417
Sequential code for the
processing of concurrent events
Find spatial and temporal
relations in an immense pool
of x/y coordinates
~25 events / second / finger
Approximative detection
GUI-Event correlation
Modularisation and composition
implementing gestures is hard
combining gestures is even harder
WHAT’S THE FUNDAMENTAL PROBLEM?
THE MIDAS APPROACH
SELECT * FROM Audience
WHERE interest > 0
Facts
Fact Base Rules
Midas
Application Layer
Core Layer
Infrastructure Layer
GUI
HardwareBridge
* Translator
FactBase
InferenceEngine
RuleBase
Shadows Model
MIDAS ARCHITECTURE
Infrastructure layer
Transform low level data into facts
Core layer
temporal and spatial operators
continuous queries via Rete algorithm
Application layer
regular program augmented with rules
MIDAS USES
(defrule PrintCursor
?c <- (Cursor)
=>
(printout t ‘‘A cursor is moving at: ’’ ?c.x ‘‘,’’ ?c.y))
MIDAS FILTERS
+
(defrule Tap
?appear <- (Cursor (finger ?f) (state ?*APPEAR*))
?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*))
(test (tShortlyAfter ?disappear ?appear))
=>
(printout t “A single tap detected at: “ ?appear.x “,” ?appear.y))
only a few lines of declarative code
MIDAS FINDS
+
(defrule Tap
?appear <- (Cursor (finger ?f) (state ?*APPEAR*))
?disappear <- (Cursor (finger ?f) (state ?*DISAPPEAR*))
(test (tShortlyAfter ?disappear ?appear))
(test (sVeryNear ?disappear ?appear))
=>
(printout t “A single tap detected at: “ ?appear.x “,” ?appear.y)
(assert (Tap (x ?appear.x) (y ?appear.y) (on ?appear.on))
(retract ?appear ?disappear))
MIDAS ENCOURAGES
(defrule DoubleTap
?tap1 <- (Tap)
?tap2 <- (Tap)
(test (tShortlyAfter ?tap2 ?tap1))
(test (sVeryNear ?tap2 ?tap1))
=>
(printout t “A double tap detected at: “ ?tap1.x“,” ?tap2.y)
(retract ?tap1 ?tap2))
small and reusable building blocks
MIDAS ENCOURAGES
(defrule FixedUpUp
?fixed <- (Fixed)
?up1 <- (Up)
?up2 <- (Up)
(test (tEqual3 ?fixed ?up1 ?up2))
(test (sLeftOf ?fixed ?up1))
(test (sLeftOf ?up1 ?up2))
=>
(printout t “Complex scroll detected!”)
(retract ?fixed ?up1 ?up2))
update 0 0,565000 0,335417
update 1 0,387500 0,450000
update 0 0,560000 0,345833
update 1 0,390000 0,462500
update 0 0,558750 0,352083
update 1 0,391250 0,479167
update 0 0,556250 0,358333
update 1 0,391250 0,489583
update 0 0,553750 0,366667
update 1 0,392500 0,502083
update 0 0,550000 0,377083
update 1 0,392500 0,516667
update 0 0,548750 0,385417
update 1 0,392500 0,531250
update 0 0,546250 0,395833
update 1 0,390000 0,543750
MIDAS CORRELATES
(defrule Gui
?tap<- (Tap)
?planet <- (Planet)
(test (sInside ?tap ?planet))
=>
(printout t “Tap inside planet!”))
add 0 0,633750 0,293750
update 0 0,635000 0,293750
update 0 0,633750 0,293750
update 0 0,633750 0,295833
update 0 0,633750 0,297917
update 0 0,632500 0,300000
update 0 0,631250 0,300000
update 0 0,630000 0,300000
update 0 0,628750 0,300000
update 0 0,627500 0,300000
add 1 0,246250 0,289583
update 1 0,247500 0,289583
update 1 0,248750 0,291667
update 1 0,250000 0,291667
update 1 0,252500 0,291667
update 1 0,256250 0,291667
Fact Base
Shadows
(GUI elements)
MIDAS SUPPORTS
(defrule nextSlide
?goRight1 <- (FlickRight (tuioId ?*RH*))
?goLeft <- (FlickLeft (tuioId ?*LH*))
?goRight2 <- (FlickRight (tuioId ?*RH*))
(test (tMeets ?goRight1 ?goLeft ?goRight2))
=>
(call PowerPoint showNextSlide)
(retract ?goRight1 ?goLeft ?goRight2))
Implemented online gestures
Templates
Kinect example
CONCLUSION
Midas multi-touch interaction framework
Declarative description of gestures (fact base and rule engine)
extensibility and reusability of gestures
correlation of GUI elements and events via shadow facts
JMidas Java integration
Ongoing work
integration of new devices: Kinect, Digital Pen & Paper, ...
extend Midas to support multimodal gestures
Rapid prototyping of multi-touch gestures
Research: http://goo.gl/OFpej - Youtube: http://goo.gl/MNDkT - Twitter: #MidasFW
Midas Demo, Tuesday @ 15.00-18.30 -- ID: 251
