Patch Panel: Enabling Control-Flow Interoperability in UbicompEnvironments

Rafael BallagasRWTH Aachen University

Department of Computer ScienceLehrstuhl fur Informatik X52056 Aachen, Germany

ballagas@cs.rwth-aachen.de

Andy Szybalski, Armando FoxStanford University

Department of Computer Science353 Serra Mall, Stanford, CA 94305

andys, fox@cs.stanford.edu

Abstract

Ubiquitous computing environments accrete slowlyover time rather than springing into existence all at once.Mechanisms are needed for incremental integration—the problem of how to incrementally add or modify be-haviors in existing ubicomp environments. Examples in-clude adding new input modalities and choreographingthe behavior of existing independent applications. TheiROS Event Heap, via its publish-subscribe coordinationmechanism, provides the foundation for interoperationthrough event intermediation, but does not directly pro-vide facilities for expressing these intermediations. ThePatch Panel provides a general facility for retargetingevent flow. Intermediations can be expressed as simpleevent translation mappings or as more complex finite-state machines. We describe an implemented prototypeof the Patch Panel, including examples of its use drawnfrom real life applications in production use in the iRoomubiquitous computing environment.

1. Introduction and Motivation

Weiser’s landmark article defined ubiquitous com-puting environments as spaces where computers areboth plentiful and subtle, allowing computation toblend invisibly into the fabric of our everyday activities[25]. Our homes [13] and office environments [10] willbe augmented with technologies to improve the way welive and work. By examining the way these buildingsevolve, it becomes clear that these ubiquitous comput-ing environments will be incrementally deployed [20].New technologies will be brought piecemeal into thesespaces [4]. Clearly, system components like physical de-vices, applications, and network services cannot be ex-

pected to anticipate every other component they mayencounter. These entities need to be able to coherentlycommunicate without a priori knowledge of each other.Moreover, the nature of the interaction must be mean-ingful to both the individual components and the usersof the system.

These constraints raise the questions: How does onedesign a system that may be augmented with futuredevices and services whose nature or feature set can-not be predicted in advance? Once the system is work-ing, how does one integrate new devices and applica-tions that may have no knowledge of each other’s exis-tence or function? We refer to this challenge as incre-mental integration, and we note that answering thesequestions would also answer the question of how to getcurrent devices and services to interoperate that werenot designed to do so.

2. Control-Flow Interoperability

The question of incremental integration has beencentral to our work in the iRoom, a conference-room-style ubicomp environment [10]. Our experience hasled us to distinguish two main modes of interopera-tion: data-flow and control-flow. Data-flow interopera-tion refers to resolving mismatches in data type, a prob-lem that received much attention in the mobile comput-ing literature [7]. For example, a user may wish to con-nect a camera that produces images in JPEG format toa printer that accepts data in PostScript format. TheiROS DataHeap [12] provides semi-transparent supportfor datatype transformation; Speakeasy [5] also pro-vides such support by exploiting mobile code “prox-ies” to perform datatype conversion.

However, datatype conversion is insufficient to en-able interoperation. Continuing the above example, theuser may want to specify color vs. black-and-white

printing, choose paper size, or the order in which pageswill print. The communication of metadata and com-mands between the camera and the printer will varydepending on the the specific devices and their capa-bilities. We refer to this challenge as control-flow inter-operability.

To date, several methods have been proposed to ad-dress control-flow operability. One method is directuser intervention via GUI’s as in iCrafter [19] andSpeakeasy [5]: each time the user wishes to print a pic-ture, a specialized GUI allows the user to set printingpreferences for that specific printer model. This ap-proach does not support making “automatic” connec-tions free of user intervention, for example, using sensordata from a motion detector to turn on the lights. An-other approach to control-flow interoperability is inter-face standardization, as attempted by Sun’s Jini [24]:each class of service or device (e.g. printers) must ad-here to a set of standard interfaces, allowing substitu-tion of any service or device in that class. This turnsout to be surprisingly difficult, even for such a seem-ingly simple class of devices as printers: after more thantwo years in committee, the Jini specification for print-ers is still (as of January 2004) labeled “pending rati-fication”. One may infer that in fact the definition ofa printer is a moving target since the feature set keepschanging. As new services or features are created, theycannot be exploited by applications until their integra-tion into existing programmatic interfaces has passedthe standards-approval process, potentially leading tounacceptable delay in adoption of the new service orfeature.

The third approach to control flow interoperability,embodied by the Patch Panel, is intermediation. Thisapproach utilizes a decoupled communication model,such as event publish/subscribe, for inter-componentcommunication. Implicit in the model is the ability tointercept and rewrite these event streams. The iRoom’score software component, the Event Heap [9], wasspecifically designed to allow for such intermediation.However, it does not directly provide any facilities forexpressing intermediation.

The Patch Panel fills this gap by providing interme-diation facilities on top of the Event Heap. In essence,it provides a generic set of mechanisms for intercept-ing and translating incoming events to outgoing events,enabling control-flow interoperability among any set ofentities that communicate via exchanging events overthe Event Heap. Previous work has shown [9] that itis very easy to connect “legacy” desktop applications(such as Win32 applications that use OLE) and Web-based applications to the Event Heap, so that theycan communicate using a common substrate; the Patch

Panel leverages this ease of integration and adds thenecessary machinery to perform the event translationsneeded to connect such components together. For therest of this paper, we will generally focus on EventHeap-aware applications, although the Patch Panel ap-proach is effective for any system using decoupled com-munication.

Systems like Context Toolkit [21] and Smart-its [3]combine information from multiple devices into ab-stracts of context cues. These context cues are essen-tially event aggregates that can be mapped to deviceor service control. Others have demonstrated the ef-fectiveness of using state machines to express aggre-gate event composition for publish / subscribe systems[15, 17]. The similar capabilities included in the PatchPanel may also be used for this purpose, but this worksets out to show that they can be used for intermedia-tion and incremental integration.

The rest of the paper proceeds as follows. We be-gin with a basic description of the intermediation fa-cilities provided by the Patch Panel. Next we describemore formally how the notation used in the basic ex-amples actually maps to the intermediation machineryavailable in the Event Heap. We then describe severalprogramming patterns that arise in addressing control-flow interoperability and illustrate how the Patch Paneladdresses each one; descriptions is are based on real-life examples from existing iRoom applications includ-ing the iClub [22] and the Workspace Navigator [8]. Fi-nally, we discuss Patch Panel deployment scenarios, in-cluding the configuration interfaces available for bothexpert/administrator users and casual users.

3. The Patch Panel and Intermediation

3.1. Review of iROS and the Event Heap

For readers unfamiliar with the Event Heap, the cen-tral coordination mechanism used in the iROS ubicompsoftware framework, we briefly review its salient char-acteristics; details of its design can be found in [9], andan overview of how it enables inter-application coor-dination in the iRoom ubiquitous computing environ-ment can be found in [10]. The Event Heap (EH) uses atuplespace abstraction to provide, among other things,event publish/subscribe to a set of clients. An eventconsists of an unordered set of attribute-value pairs,one of which is designated as the event’s type and theremainder of which are application-specific. The ba-sic operations are put, which posts an event, and get,which queries for the existence of an event based on atemplate that specifies required fields and constraintson their values. Event subscription is also provided:

Light Controller

Projector Controller

iPong Application

Wireless ButtonProxy

Joystick Proxy

Patch Panel

Event Heap

Patch Panel

Put:Button

Get:Light

Put:

Joystick

Get

:Bu

tton

Get:

Joystick Get:iPong

Get:Projector

Put:

iPon

g

Put:Projector

Put:

Light

Figure 1. The Patch Panel adds a level of indirection to the communication channel to two components inorder to perform event intermediation. The publish/subscribe semantics are also demonstrated. Note thatthere is actually only one Patch Panel process per Event Heap; two are shown above for visual clarity.

clients can receive a callback when events matchinga particular template are posted to the EH. In keep-ing with the Boundary Principle [14], a single EH isthe locus of interaction for a single ubiquitous comput-ing environment, and a service or device can partici-pate in that environment if and only if it can commu-nicate with that environment’s EH. Various librariesand other software components allow EH clients to bewritten in Java, C/C++, Visual Basic, Perl, Python,and other languages; servlets allow Web-based clientsto perform limited EH operations as well.

3.2. Event Translations as Mappings

The Patch Panel enables intermediation among enti-ties that communicate via event publish/subscribe (SeeFig. 1). Simple intermediations are expressed as map-pings that connect “triggers” (observed events emittedby some entity) with “outputs” (new events emitted asa result of the trigger, presumably for consumption bya different entity). The Patch Panel works by subscrib-ing to all events. If an event is received that matches atrigger condition, the Patch Panel generates the corre-sponding output event(s) and posts them to the EH.

A mapping (trigger → output events) is the basicabstraction provided by the Patch Panel. For example,suppose we have a wireless iStuff button [1] that posts aButton event1 each time it is pressed with string-valuedfield id, a light dimmer service that responds to Lights

1 The stylized event names such as Button and Lights representthe mandatory event type field of the events.

events containing an integer-valued field brightnessbetween 0 and 10, and a projector-control service thatresponds to events Projector with boolean-valued fieldpowerOn. We can configure the button to turn the lightsand projector on by establishing the Patch Panel map-ping in Figure 2.

Now, when someone presses the button, the follow-ing sequence of operations occurs:

1. In response to being pressed, the button posts aButton(id=red) event to the EH.2

2. Although neither the light service nor the projec-tor service recognize the Button event, the PatchPanel recognizes it as a trigger for the Button map-ping shown above.

3. The mapping fires, causing the Patch Panel topost the events Lights(brightness=10) and Pro-jector(powerOn=true) to the EH.

4. The light controller recognizes the Lights eventand turns the lights on; the projector controllerrecognizes the Projector event and turns the pro-jector on.

3.3. Intermediation as State Machines

Suppose instead that we want the red button to tog-gle the lights and the projector between on and off. As-suming (as is often the case) that we cannot “query”

2 As described in [1], simple hardware devices such as buttonsand sliders typically communicate with a software proxy thatconnects the devices to the EH.

Button(id=red) → Lights(brightness = 10), Projector(powerOn=true)

Figure 2. Simple Patch Panel mapping that turns on lights and projector

the light service or the projector service to inquireabout their current state (on or off), we could turn thebutton into a toggle switch by using a finite state ma-chine (FSM) such as in Fig. 3.

If the red button is pressed while the Patch Panel isin the Off state, it will turn on the lights and the pro-jector, and vice versa. Although the button is state-less and the light and projector services do not exposetheir internal state, the Patch Panel instantiates anFSM that “remembers” the current state of the inter-action and responds accordingly to the Button(id=red)trigger event.

4. FSM’s and Mappings

In this section we describe how the state machinesused in the foregoing examples are actually translatedto event mappings, and how the mappings are modi-fied on each state transition. We also discuss the PatchPanel’s performance and fault-tolerance.

4.1. Mappings as State Machines

As illustrated by Figure 3, the effect of expressing in-teractions as FSM’s is that the mapping of a given trig-ger to a set of output event(s) depends on the currentstate of the FSM. In fact, such interactions are imple-mented using the same underlying mechanism that isused for simple mappings like the Button examples insection 3.2. To illustrate how this is done, consider amore complex example. Here we add support for an in-room motion sensor that posts a MotionSensor eventwhenever it detects activity. If no motion is detectedfor timeout seconds, the lights and projector turn offautomatically. The FSM description is shown in Fig-ure 4.

Using Figure 4, we can formalize the Patch Panel’soperation as follows. Let S0, . . . , Sn−1 be the n statesof an FSM that expresses a Patch Panel-mediated in-teraction. From Figure 4, let S0 be the Off state and S1

the On state. Let mi be a simple event mapping of theform ti → oi1, oi2, . . . oik, where ti is a trigger event andoi1 . . . oik are the output events triggered by ti. Finally,let NS(i) be the number of the next state to go to af-ter emitting the output events of mapping mi. FromFigure 3, each of the “on” clauses generates one map-ping mi and one next-state NS(i) as demonstrated inFigure 5.

Let Mi be the set of mappings consistent with theFSM being in state Si. For Figure 4, we would haveM0 = {m0} and M1 = {m1,m2,m3}. The PatchPanel maintains a single set P of mappings that is cur-rently active, i.e. against which incoming events willbe checked for triggers. When a state transition oc-curs into state Si, P must be set to Mi. With this no-tation, we can express in pseudocode the operation ofthe Patch Panel (by convention, S0 is the initial state):

1. Compute all mi, NS(i), and sets M0 . . .Mn−1

from textual FSM description;

2. Set P to M0;

3. do forever:

(a) wait for an event ti that triggers some map-ping mi in P ;

(b) emit the mapping’s output eventsoi1, . . . , oik;

(c) set P to MNS(i);

In other words, the Patch Panel’s textual front-endtakes a description of a Mealy-style FSM and gener-ates the corresponding per-event mappings to imple-ment that FSM. As we will show, therefore, the PatchPanel can be configured either by modifying individ-ual mappings using a GUI editor or by creating a tex-tual FSM description such as those above. The FSM de-scription language is loosely based on the intermediate-file format used by fsmc, a finite-state machine com-piler used in several digital design courses. fsmc con-verts FSM specifications to sum-of-products Booleanexpressions that can be used to program logic arrays.The only control-flow keywords we provide are stateto declare a new state, on to specify trigger events,send to emit an output event, and goto to transitionto the next state. A complete description of the lan-guage can be found in [2].

We now discuss the idiosyncrasies of running anFSM engine on top of the Event Heap.

4.2. PPMapping Events

The actual modification of the set P referredto above is itself controlled by events. In particu-lar, the Patch Panel looks for a special event typecalled PPMapping. These events can be used to add, re-move and modify the active mappings P .

state Off {on Button(id = red) {

Lights(brightness = 10);Projector(powerOn = true);goto On;

}}state On {

on Button(id = red) {Lights(brightness = 0);Projector(powerOn = false);goto Off;

}}

state On

state Off

on Button(id=red)/send

Lights(brightness=0) &Projector(powerOn=false)

on Button(id=red)/sendLights(brightness=10) &Projector(powerOn=true)

Figure 3. Simple light-and-projector toggle

state Off {on Button(id = red) { // turn things on manually

Lights(brightness = 10);Projector(powerOn = true);goto On;

}}state On {

on MotionSensor { // motion detected, so...set timer 1000*timeout; // ...reset idle timergoto On;

}on timer { // no motion sensed, so...

Lights(brightness = 0); // ...turn things offProjector(powerOn = false);goto Off;

}on Button(id = red) { // turn things off manually

Lights(brightness = 0);Projector(powerOn = false);goto Off;

}}

Figure 4. FSM description with timers

m0 = Button(id=red) → Lights(brightness=10), Projector(powerOn=true) NS(0) = S1

m1 = MotionSensor → set timer 1000*timeout NS(1) = S1

m2 = timer → Lights(brightness=0), Projector(powerOn=false) NS(2) = S0

m3 = Button(id=red) → Lights(brightness=0), Projector(powerOn=false) NS(3) = S0

Figure 5. Mappings that implement the state machine for in-room motion sensor

The reason for exposing PPMapping as an event-based interface (i.e. as an external primitive) is to al-low dynamic reconfiguration: it allows the set of activemappings to be changed not only as a result of inter-nal state transitions, but also by an external controllerprogram. In section 5 we illustrate an example of howto exploit this capability in the user interface for pro-gramming and using the Patch Panel.

4.3. Atomic State Transitions

By design, the Event Heap does not guarantee or-dering of events from different sources. Therefore, out-put events from the PatchPanel and trigger events fromother sources may be interleaved. This is problematicsince the FSM abstraction requires that the emittingof output events (step 3b above) and transition to thenext state (step 3c) must occur atomically. To addressthis, the Patch Panel provides event chains, which aregroupings of events that must be processed atomically.In our current implementation, any outgoing event con-taining a field ChainEvent is considered to be part ofan atomic event chain. The Patch Panel’s event emit-ter passes event chains directly to its own event han-dling loop (i.e. these events do not travel through theEvent Heap at all), where they are given highest prior-ity.

4.4. Performance and Failure Semantics

The Patch Panel is often in the critical path ofthe human interface action-perception loop. The ba-sic measure of Patch Panel performance is the delaybetween the posting of a trigger event to the EventHeap and the receipt of the correct resulting out-put event(s) to the Event Heap. Our microbenchmarkfor simple mappings in Figure 6 shows this delay tobe about 12ms, of which 6ms is due to the PatchPanel itself and the remainder is attributed to seri-alization/deserialization of events, network delay, andinternal Event Heap delay. The total delay increaseswith events that contain more fields. It may also in-crease when there are many (over 100) mappings, be-cause the Patch Panel’s mapping hash table was de-signed for small numbers of mappings.

Because the Patch Panel is an Event Heap client, itinherits the Event Heap’s robustness [18] in handlingfailures. In particular, the failure of other Event Heapclients does not compromise the operation of the PatchPanel. Failure of the Patch Panel causes intermediationto cease until it is restarted. The Patch Panel’s inter-nal mappings are saved to disk, and the states of the

active state machines will survive restarts of both theEvent Heap and the Patch Panel.

5. Functionality Motivated by Example

One of the lessons of our work relates to the reper-toire of “programming patterns” needed to cover abroad variety of incremental-integration scenarios. Inthis section, we describe some of the patterns supportedby the Patch Panel, motivating each one with a real-life example. As a preview, the mechanisms we will de-scribe are as follows:

1. Allowing for dependent translations, where thefield values of output events are not known at thetime the mapping is specified. Instead, the outputfields must be derived from field values of triggerevents at run time.

2. Programmatically exposing to applications theability to change mappings on the fly, allow-ing the construction of GUIs and other applica-tions whose function is to configure the PatchPanel itself.

3. Allowing the use of global variables (whose valuepersists across individual firings) to further sup-port interactions requiring persistent state.

4. Providing an abstraction for time, allowing fortime-based interactions

This set of examples is based on two real applica-tions: the iClub [22], an iRoom application that cre-ates an interactive dance club environment using thelarge displays in the iRoom, and Workspace Navi-gator (WSN) [8], an application that captures meet-ings in the iRoom using video cameras, digital white-board systems, screen capture software and a docu-ment archive. Videos showing each application in useare at http://iwork.stanford.edu.

iClub is a distributed application that includes aplaylist program to select songs and sound effects, anaudio proxy application that plays the audio and pub-lishes a “beat clock” event synchronized to each beat ofthe music, several visualization applications that syn-chronize to the beat of the music by subscribing to thebeat clock events, and a GUI that controls various as-pects of the music as it plays

5.1. On-the-Fly Integration

The audio proxy recognizes events to adjust the vol-ume and tempo of the music, inject sound effects, andapply high- and low-pass frequency filters (a commonaudio special-effect) to the music. In the original ver-sion of iClub, a human DJ would use a GUI to control

0

10

20

30

40

50

60

70

80

0 100 500 1000

Translation Round Trip Time (ms)

# of fields

# of mappings

Figure 6. Patch Panel performance degradation. The performance degrades as the number of oustandingmappings increases, and as the number of event fields increases.

Figure 7. (Left)the iClub in Action with iSlider, (Right)examples of iStuff [1] input devices

these aspects of the music. However, an observer sug-gested that the clubgoers themselves should be able toparticipate in the music creation without leaving thedance floor. We arranged to give each clubgoer a wire-less button with a unique ID as they entered the room.Each button was mapped through the Patch Panel todirect the audio proxy to play a sound, so each clubgoerhad her own characteristic sound effect that could beinjected by pressing her button. We refer to this com-mon programming pattern—connecting a new physicalor other UI to an existing behavior—as on-the-fly inte-gration.

5.2. Range Normalization

The developers’ next inspiration was to allow a DJto use a wireless handheld slider to have mobile con-trol the tempo of the current song. Conceptually, thisis similar to the previous example in that a slider event

must be used to trigger an iClubAudio output event,except that the value of the slider must also be re-flected in the output event. Furthermore, the specificslider we used produces real numbers in the range 0.0 to1.0, whereas the iClubAudio’s tempo parameter mustbe an integer from −10 to +10. We use the term rangenormalization to describe this pattern for incrementalintegration. The Patch Panel supports range normal-ization by allowing output event fields to reference in-put event fields and by providing a simple arithmeticexpression evaluator. Figure 8 shows a mapping thatconnects the slider to the tempo control.

This mapping will fire when the Patch Panel receivesan event of type iSlider that contains a field namedposition. The tempo field of the iClubAudio outputevent is computed from the position field of the in-put event (in.) each time the mapping fires. Similarly,the prefix out. could be used to reference other fieldsin the current output event; the Patch Panel automat-

iSlider(position = *) → iClubAudio(tempo = (int)in.position * 20 - 10)

Figure 8. Range Normalization and Equation Specification

ically detects syntax errors or output field dependencyloops and leaves such equations unresolved (in stringform) for debugging purposes.

5.3. Dynamic Reconfiguration

The wireless slider works well to control a single pa-rameter, but in order for a DJ or clubgoer to abandonthe desktop GUI completely, she must have the abil-ity to change any of the music parameters, not just thetempo. We constructed a new device by physically at-taching four wireless buttons (call them 1, 2, 3, 4) tothe slider. Pressing a button determines which musicparameter—tempo, volume, high pass filter, low passfilter—is controlled by the slider. This case is more sub-tle, because the effect of pressing a button is not toemit a new audio event, but rather to affect the han-dling of future iSlider events. In other words, pressing abutton should change the currently-active iSlider map-ping. We therefore refer to this pattern as dynamic re-configuration.

As explained previously, the Patch Panel consumesevents of type PPMapping to modify the set of cur-rently active mappings. With this in mind, the map-pings to implement the “multi-slider” handheld musiccontroller are shown in Figure 9.

When the Patch Panel receives a Button(id=1)event, for example, it emits a PPMapping event thatwill set up a new mapping for future iSlider events tocontrol the music’s tempo. This powerful dynamic re-configuration allows complex interactions to be synthe-sized without direct user interaction.

Since the above mappings are somewhat difficult tointerpret for a programmer unfamiliar with the Patch-Panel, we could also have expressed the multi-sliderexample using the FSM representation shown in Fig-ure 10, and in fact, submitting this FSM description tothe Patch Panel’s FSM compiler would effectively re-sult in the mappings in Figure 9.

Several points about this example should be empha-sized. First, the handheld DJ device and the interactionwith the iClub Audio Proxy were assembled from de-vices and services that had no a priori knowledge ofeach other. In other words, the slider has no concept ofa button or vice versa, and the iClub Audio Proxy hasno concept of any physical devices. Also in this exam-ple, the buttons and mappings, once initially set up byan administrator, represent a tangible UI for reconfig-

uring the Patch Panel that can be used by people whohave no technical skill or knowledge of Event Heap op-eration. We return to the issue of how different usersmight actually use the Patch Panel in section 6.

5.4. Semantic Mismatches and Globals

At one point, the wireless slider malfunctioned, andalthough we did not have another slider immediatelyavailable, we did have a joystick. However, while aslider’s position naturally maps to the value being con-trolled, a joystick’s position naturally maps to the rateof change of the value being controlled (since joysticksare self-zeroing). We refer to this circumstance as se-mantic mismatch, and in this case can be resolved byusing an FSM with global variables as shown in Fig-ure 11. Global variables store values that must per-sist across firings of mappings; variables can be set us-ing “PPVariable” events and dereferenced on the out-put side of any mapping by using the prefix global.To mimic the slider’s behavior with a joystick, we use aglobal variable to hold the current “slider position” andadjust the global variable’s value each time the joystickis moved. To avoid wild fluctuation in the variable’svalue, we can use timers (as previously described) tocontrol the interval at which the joystick is sampled;changing the timer value changes the sensitivity of thejoystick as a controller. Although the joystick is an im-perfect interaction modality for the iClub Audio Proxy,the Patch Panel made it possible to use it as an ade-quate substitute until the slider could be replaced.

5.5. Integration in Workspace Navigator

The last example concerns Workspace Navigator(WSN) [8], an application that captures multi-personmeetings in interactive rooms containing shared pub-lic displays. WSN’s GUI provides a “bookmark” fea-ture that allows a participant to flag an importantmoment in the meeting; WSN’s meeting-replay toolscan then be used later to reconstruct the state of themeeting (e.g. which documents were visible on eachof the shared displays) at the time the bookmark wasinserted. To implement the bookmark function, theWSN GUI console sends a Bookmark event to the WSNserver application when the Bookmark GUI widget isclicked.

Button(id=1) → ( PPMapping[ iSlider(position = *) → iClubAudio(tempo=(int)in.position*20-10) ] )Button(id=2) → ( PPMapping[ iSlider(position = *) → iClubAudio(volume=(int)in.position*100) ] )Button(id=3) → ( PPMapping[ iSlider(position = *) → iClubAudio(highfreq=(int)in.position*100) ] )Button(id=4) → ( PPMapping[ iSlider(position = *) → iClubAudio(lowfreq=(int)in.position*100) ] )

Figure 9. Patch Panel mappings that enable the multi-slider handheld music controller

state ControllingTempo {on iSlider(position=*) { send iClubAudio(tempo=int(in.position)*20-10); }on Button(id=2) { goto ControllingVolume; }on Button(id=3) { goto ControllingHighFreq; }on Button(id=4) { goto ControllingLowFreq; }

}state ControllingVolume {on iSlider(position=*) { send iClubAudio(volume=int(in.position)*20-10); }on Button(id=1) { goto ControllingTempo; }on Button(id=3) { goto ControllingHighFreq; }on Button(id=4) { goto ControllingLowFreq; }}

}...

Figure 10. FSM description of iClub multi-slider mappings

During user testing of WSN, one user complainedthat inserting a bookmark required disrupting themeeting to acquire the shared keyboard and mouse(in order to interact with the GUI), discouraging usersfrom taking advantage of this feature. The researcherproposed giving each meeting participant a wirelessbutton that could be discreetly pressed to add a book-mark during the meeting. This approach has the addi-tional benefit that each bookmark could be associatedwith the participant who inserted it.

To implement this, an iRoom administrator createda simple Web-based “This Is My Button” wizard thatconfigures an iButton (an iStuff wireless physical but-ton) to send Bookmark events. On entry to the iRoom,a meeting participant, say Rachel, picks up an iBut-ton from a bucket of buttons, enters her name into theWeb form, and submits the form. The form submissionruns a servlet that waits for the next button press fromany iButton. Rachel now presses the (physical) button,causing the servlet to establish a Patch Panel mappingconnecting her particular button to bookmark eventswith her name attached to them.

This is another example of dynamic reconfiguration.The entire process of integrating the wireless buttoninteraction into the WSN project only took about anhour and did not require any changes to the Workspacenavigator code or the wireless buttons.

5.6. Summary

It has been about a year since work began on thePatch Panel, and in that time, researchers in our spacehave been using it to put together separate compo-nents in ad-hoc ways. Uses have ranged from enhanc-ing existing applications, as illustrated by the foregoingexamples, to creating new interactions using physi-cal input devices. All the examples above are real andare in production use. Although each is simple by it-self, together they illustrate the versatility and relativesimplicity of the Patch Panel approach to interoper-ation, allowing multi-device systems to be deployedor existing applications to be augmented in a mat-ter of minutes. The table in Figure 12 summarizesthe examples, including the interoperability program-ming pattern illustrated by each and the Patch Panelmechanism(s) leveraged to achieve the desired re-sult.

6. Configuration and Ease of Use

The ease of configuring mappings for interoperationis a key consideration in evaluating the usefulness ofthe Patch Panel. Since Patch Panel mappings can bechanged by any client that emits PPMapping events,

state JoystickMoved {on Joystick(joystickX, joystickY) {

global.currentX += in.joystickX * global.scaleFactor;global.currentY += in.joystickY * global.scaleFactor;send Position(global.currentX, global.currentY);set timer sampleRate;goto WaitingForSample;

}}state WaitingForSample {

on timer { goto JoystickMoved; }}

Figure 11. Resolving semantic mismatch between relative-position and absolute-position devices. We haveminimally stylized the code for readability.

(Section) Example Name On-the-flyIntegrationDynamic ReconfigurationState MachineRepresentationEquations / RangeNormalizationGlobal Variables /Semantic MismatchTimers(3.2) Lights and Projector on x

(4.1) Lights and Projector toggle x x x

(4.1) Motion sensor toggle x x x x x

(5.1) iClub sound buttons x

(5.2) iClub slider x x

(5.3) iClub multi-slider x x x

(5.4) iClub joystick x x x x x x

(5.5) Workspace Navigator x x

On-

the-

flyIn

tegr

atio

nD

ynam

icR

econ

figur

atio

nSt

ate

Mac

hine

Rep

rese

ntat

ion

Equa

tions

/ R

ange

Nor

mal

izat

ion

Tim

ers

Glo

bal V

aria

bles

/

Sem

antic

Mis

mat

ch

Figure 12. Patch Panel Example Summary Table

we can create a variety of configuration interfaces, fromnarrowly-specialized GUI’s for casual users to sophis-ticated GUI’s or textual interfaces for administratorsand expert users.

6.1. Support for Expert Users

We distinguish expert from casual users. Expertusers include: developers who want to create or aug-ment applications by connecting new behaviors via thePatch Panel; system administrators; and “power users”(akin to those who can write complex macros in spread-

sheet programs) who want to modify the behavior ofexisting applications.

The FSM scripting language described in section 4.1provides a powerful textual front-end for expert users.We have also built the Patch Panel Manager (fig-ure 13), a tree-based view of the Patch Panel map-pings that allows expert users to graphically browsethe current mappings and create new ones. Using thisGUI requires some understanding of Event Heap se-mantics as well as the event interfaces of individualcomponents. In everyday situations, the Patch Panel

Figure 13. The Patch Panel Manager

→ →

Figure 14. Button-to-URL Configuration Servlet

Manager is used to visualize and debug the state ofthe Patch Panel. It is also particularly useful for mak-ing on-the-fly modifications to mappings or for “debug-ging” in-progress mappings; for example, changing theamplification factor applied to an input device.

6.2. GUI’s for Casual Users

While the Patch Panel Manager GUI and the FSMscripting language satisfy most of an administrator’sPatch Panel needs, it is also important for casual usersof different skill levels to be able to define interactionsto meet their needs. To demonstrate the potential sim-plicity of configuring the Patch Panel we have created aButton-to-URL Configuration Servlet (Fig. 14). It pro-vides a web-based “wizard” interface that configuresan iStuff button to display a user-chosen Web page. Toconfigure a button, the user types the target URL intothe configuration Web page and clicks “Submit”. Thenthe servlet probes the Event Heap for the next but-ton press. The user picks up any button from a bucketof buttons, and presses it. The servlet associates thepressed button with the URL; any subsequent eventsfrom that button will be translated to an event thatopens the URL on a large public display.

This configuration servlet and the iClub multi-sliderillustrate a class of interfaces that require developer ef-

fort up front to expose specific and limited Patch Panelfunctionality to less technical users. We are in the pro-cess of exploring more general “wizard” type configu-ration interfaces for casual users of iStuff [1].

7. Conclusions

We have argued from the beginning [18] that incre-mental integration is an essential property of robustly-evolving ubicomp environments, and that a powerfultechnique to enable integration is by intermediationof control flow. This insight was originally expressedin [15], but to the extent that intermediation is a solu-tion in search of a problem, we believe that interoper-ation and evolution in ubiquitous computing environ-ments is that problem.

Although an expert user can certainly write a full-blown Java program with arbitrarily complex eventtranslation logic, we hypothesized that a small num-ber of programming patterns serve to capture a widevariety of common integration and evolution tasks.Our goal was to provide a simple programming sub-strate that supports rapid and easy expression of themost common such patterns through its textual in-terface, and admits of the rapid creation of graphi-cal or physical interfaces for novice users to perform

narrowly-specialized configuration tasks, as the iSliderand Workspace Navigator examples illustrate.

The result of our work to date is a system that en-ables intermediation-based control-flow interoperabil-ity in ubiquitous computing environments. Semanticmismatches between incompatible interfaces can be re-solved with state machine and equation evaluation ca-pabilities. The dynamic reconfiguration feature of thePatch Panel is critical in enabling state machine ca-pabilities, retargeting event flow based on user inputor context from sensor data, and enabling simple userinterfaces to reprogram the Patch Panel. This combi-nation of features provides a powerful integration toolthat can be used to easily create and modify, often ina matter of minutes, interactions between networkedhardware and software components in ubiquitous com-puting environments.

8. Acknowledgments

The authors thank Maureen Stone, Brad Johanson,Jan Borchers, and Shankar Ponnekanti for insightfulcomments and suggestions for this work. This researchis supported by a National Science Foundation Gradu-ate Research Fellowship and by a grant from the Wal-lenberg Global Learning Network. Any opinions, find-ings, conclusions or recommendations expressed in thispublication are those of the authors and do not neces-sarily reflect the views of the funding agencies.

References

[1] Ballagas, R., Ringel, M., Stone, M., Borchers, J: iStuff: aPhysical User Interface Toolkit for Ubiquitous Comput-ing Environments. Proc. of CHI (2003)

[2] Ballagas, R., Szybalski, A., Fox, A.: ”Overview of thePatch Panel Finite State Machine Description Lan-guage.” Stanford Computer Science Technical Report(2003)

[3] Beigl, M., Gellersen, H. “Smart-Its: An Embedded Plat-form for Smart Objects”, Smart Objects Conference(2003).

[4] Edwards, K., Grinter, R.: At Home with UbiquitousComputing: Seven Challenges. Proc. of UBICOMP(2001) 256-272.

[5] Edwards K., Newman, M., Sedivy, J.: The Case for Re-combinant Computing. Xerox Palo Alto Research Cen-ter Technical Report CSL-01-1 (2001)

[6] Edwards K. et al: Challenge: Recombinant Comput-ing and the Speakeasy Approach. Proc. of MOBICOM(2002)

[7] Fox, A., Gribble, S.D., Chawathe, Y., Brewer, E.A.:”Adapting to Network and Client Variation Using Ac-tive Proxies: Lessons and Perspectives”. IEEE PersonalComm. (1998)

[8] Ionescu, A., Stone, M., and Winograd, T. “Workspace-Navigator: Capture, Recall, and Reuse using SpatialCues in an Interactive Workspace.” Stanford ComputerScience Technical Report 2002-04 (2002).

[9] Johanson, B. and Fox, A. The Event Heap: A Coordina-tion Infrastructure for InteractiveWorkspaces.Proceed-ingsof the4th IEEEWorkshoponMobileComputerSys-tems and Applications (WMCSA-2002).

[10] Johanson, B., Fox, A., Winograd, T.: The InteractiveWorkspacesProject:ExperienceswithUbiquitousCom-puting Rooms. IEEE Pervasive Computing Magazine,Vol. 1, Issue 2, (2002) 71-78.

[11] Johanson, B., Hutchins, G., Stone, M., and Winograd,T. PointRight: Experience with Flexible Input Redirec-tion in Interactive Workspaces. Proc. UIST (2002).

[12] Kıcıman, E., Fox, A., Using Dynamic Mediation to Inte-grate CTOS Entities in a Ubiquitous Computing Envi-ronment. Second Annual Symposium on Handheld andUbiquitous Computing (2000)

[13] Kidd, C. D., Orr, R. J., Abowd, G. D., Atkeson, C. G.,Essa, I. A., MaclIntyre, B., Mynatt, E., Starner, T. E.,Newstetter, W.: The Aware Home: A Living Laboratoryfor Ubiquitous Computing Research. Proc. of CoBuild.(1999)

[14] Kindberg, T., Fox, A., System Software for UbiquitousComputing. IEEE Pervasive Computing 1(1).

[15] Munson,M:SystemSupport forComposingDistributedApplicationsUsingEvents.Dissertation.PembrokeCol-lege, University of Cambridge (1998)

[16] Myers, B. Kosbie, D.: Reusable Hierarchical CommandObjects. CHI 1996.

[17] Pietzuch, P., Shand, B., Bacon, J. A Framework forEvent Composition in Distributed Systems. In Proc.Middleware (2003) 62-82.

[18] Ponnekanti, S., Johanson, B., Kiciman, E., and Fox,A. Portability, Extensibility and Robustness in iROS.In Proc. IEEE International Conference on PervasiveComputing and Communications (PerCom 2003), Dal-las, TX, March 2003.

[19] Ponnekanti, S., Lee, B., Fox, A., Hanrahan, P., Wino-grad, T.: iCrafter: A Service Framework for UbiquitousComputing Environments. Proc. of UBICOMP (2001)

[20] Rodden,T., Bedford, S.: TheEvolution ofBuildings andImplications for the Design of Ubiquitous ComputingEnviroments. Proc. of CHI. (2003) 9-16

[21] Salber, D.,Dey, A., Abowd, G. “The Context Toolkit:Aiding the Development of Enabled Applications.”Proc. of CHI (1999) 434-441.

[22] Samberg, J., Fox, A., Stone, M. “iClub, An InteractiveDance Club”, Ubicomp 2002 Video Program (2002).

[23] Taylor, R., et al. A Component- and Message-Based Ar-chitectural Style for GUI Software. IEEE Transactionson Software Engineering, (1996)

[24] Waldo, J.: Alive and Well: Jini Technology Today. Com-puter Vol. 33, Issue 6. (2000) 107-109

[25] Weiser, M.: The Computer for the 21st Century. Scien-tific American, Vol. 265, Issue 3. (1991) 91-104.