about interestate

8/12/2019 about interestate

1/10

1

InterState:Interaction-OrientedLanguagePrimitivesforExpressing GUI Behavior

Stephen Oney, Brad MyersCarnegie Mellon University

Joel Brandt

Adobe Research

ABSTRACT

The event-callback programming model used by most GUI

development frameworks frequently leads to disorganized

and hard-to-maintain code. We believe there is significant

value in designing language primitives that expressly ad-

dress the challenges of writing and re-using GUI code. To

explore this, we created InterState, a language and pro-gramming environment for expressing interactive behaviors

in graphical applications. With InterState, programmers

express behaviors declaratively as combinations of states

and constraints. InterState introduces new primitives for

defining and re-using behaviors, a visual notation for these

primitives, and a live editor. We conducted a laboratory

study to evaluate InterStates usability and found that par-ticipants were faster at understanding and modifying GUI

components written in InterState than in event-callback

code. Additionally, to better understand InterStates scope

and expressiveness, we used it to implement a series of full-

featured interfaces.

INTRODUCTION

Nearly all widely-deployed user interface (UI) frame-workse.g., Cocoa, QT, Java Swing, and JavaS-

cript/HTMLrely on an event-callback programming mod-

el [18]. By requiring programmers to express interactive

behavior as a series of imperative callbacks, this program-

ming model tends to produce error-prone spaghetti code

because it is necessary to split the implementation of a sin-gle behavior across many locations in the source [21,28].

To address this issue, researchers and practitioners have

created libraries that augment existing languages and UI

frameworks with new programming models, including con-

straints (relationships that are maintained automatically)

[17,21,24] and state machines [1,28]. However, when pro-ducing a UI is the programmers primary goal, many as-

pects of the underlying languagewhich evolved around

computation-oriented softwarecan be hindrances [16].

Our new development system, named InterState, is based

on the insight that by redesigning many language and

runtime featuresincluding the event-callback framework,

inheritance, the syntax, and the edit-compile-run evaluation

cyclewe may be able to create a development tools more

suitable for UI development. InterState is intended to be a

language for specifying UIbehavior and appearance, rather

than a general-purpose programming language. It includes a

mechanism for communicating with back-end code written

in other languages, allowing developers to connect a front-

end written in InterState with a back-end written in another

language.

Overview of the InterState Approach

Two of the most difficult aspects of programming interac-

tive behaviors are dealing with states and expressing con-

straints [16,27,28]. ConstraintJS [24] addressed these diffi-

culties by allowing developers to express interactive behav-

iors in JavaScript and HTML/CSS as constraints that are

enforced only in particular states. InterState extends theideas behind ConstraintJS in two primary ways. First, new

language primitives for inheritance and templating increase

the expressiveness of constraints and states. Second, a visu-

al notation and a live editor remove the edit-compile-run

cycle, enabling immediate evaluation and allowing devel-

opers to see objects state.

An important innovation in InterState is a new form of in-

heritance. Developers often want to re-use, combine, and

inherit behaviors. However, the traditional notion of inher-

itance in languages like Java or JavaScript only allows

properties and methods to be inherited. InterState introduc-

es a style of inheritance that extends traditional prototype-

instance inheritance mechanisms to allow behaviors to beinherited. It does this by allowing objects to inherit not only

properties and their constraints but also their prototypes

state machines. InterState also introduces a mechanism for

templatingthat allows interactive components to be dynam-

ically created and updated to reflect changes in an underly-

ing data model.

It is also important that users can understand their pro-

grams state. Thus, we introduce a visual notation for our

Figure 1.A screenshot of a basic InterState object, named rect.

Properties, which control rects display, are represented as rows

(e.g. x, y, and fill). States and transitions are represented as

columns (e.g. idle and dragging). An entry in a propertys row

for a particular state specifies a constraint that controls that prop-

ertys value in that state. Here, while rect is in the dragging

state, x and y will be constrained to mouse.x and mouse.y re-

spectively, meaning rectwill follow the mouse while dragging!

Submitted to CHI 2014


2/10

2

language. In most languages, understanding what user

events affect a particular property or, conversely, what

properties are affected by a particular user event, can be

difficult because the code is often spread throughout multi-

ple locations. InterState displays object properties as rows

and states as columns, as illustrated in Figure 1. This layoutallows developers to see which events affect a property by

scanning across the propertys row and which properties anevent affects by looking at the events column. It also re-

moves much of the boilerplate required to express con-

straints in other systems, allowing users to express con-

straints with simple equationslike those in spreadsheets

rather than requiring inline functions [17,24].

Because the primary goal of most user interfaces is to be

usable, its also important that developers can immediately

use and evaluate their application as they create it, also

called reflection-in-action [29]. To enable reflection-in-

action, we implemented a live visual editor, meaning thatchanges in the editor are immediately reflected in the run-

ning application and changes in the runtime state are imme-

diately visible in the editorContributions

This paper makes four primary contributions:

Language primitives for expressing UI behavior thatbuild on the idea of combining states and constraints,

a specification for inheritance and templating that al-lows these primitives to be re-used across objects,

a visual notation and live editor that allow developersto see the state of their program, and

an evaluation of these features through a laboratorystudy and a series of example applications.

RELATED WORK

InterState is influenced by work in multiple domains.

The Difficulties of Developing Interactive Applications

Previous research has investigated why developing interac-

tive applications is particularly challenging. One factor isthat interaction-oriented development tools must support

different state and reuse mechanisms than general purpose

development tools [16]. InterState addresses these issues by

treating state and behavior reuse as primitives. Two moti-

vating studies found that designers think about relationships

between graphical objects with state, constraint, and event-

based concepts [27] and that reflection-in-action is im-

portant for designers creativity [25]. Thus, InterStates

basic primitives were built around dealing with states and

supporting constraints, and we implemented InterState as alive editor to support reflection-in-action.

Spreadsheet Programming

Spreadsheets are considered by many researchers to be the

most popular form of programming [18]. Part of theirappeal lies in their beginner friendliness: the user always

has a working program and errors can be localized.

There has been research in ways to use spreadsheet-like

ideas to create graphical interfaces. One such system is

Forms/3 [4], which demonstrated that procedural and data

abstractions and graphical output were viable with spread-

sheets. However, Forms/3 does not offer explicit support

for dealing with state. By including state as a languageprimitive, InterState becomes more expressive and better

suited to writing interactive behaviors, which are oftenstate-oriented.

Constraint Libraries for Imperative Languages

A number of libraries intended to simplify the development

of interactive behaviors have been written for imperative

languages. Of these, the most relevant to InterState is Con-

straintJS, a JavaScript library on which InterState is built

[24]. Like InterState, ConstraintJS combines states and con-

straints to define interactive behaviors. However, Con-

straintJS was built specifically to fit into the context of im-

perative programming with JavaScript and HTML/CSS,

whereas InterState uses a visual notation to allow users to

express interactive applications with little or no imperative

code. Achieving this goal went beyond simply adding avisual notation on top of ConstraintJS; it also required de-

signing several new language primitives.

There are several other data-binding and constraint librar-

ies, including Knockout (knockoutjs.com), Ember (em-

berjs.com), Amulet [20], Kaleidoscope [6], and D3[3]. Ad-

ditionally, FlapJax [17] provides both a constraint mecha-

nism and event model. While all of these libraries can be

effective in allowing skilled developers to write clearer

code for interactive applications, none of them include

primitives for dealing with state or a visual notation.

Finite State Machine Libraries for Imperative Languages

Since interactive applications are highly state oriented, Fi-

nite State Machine (FSM) libraries seem particularly suita-ble for GUIdevelopment tools [23]. TKZink, IntuiKit [15],

and HsmTk [2] use state machines to allow interface de-

signers to specify different application appearances in dif-

ferent states but do not allow developers to specify custom

interactive behaviors. Similarly, SwingStates [1] integrates

finite state machines with the Java Swing toolkit but does

not provide primitives for interaction or user event handling

as InterState does.

Interactive and Live Editors

Many programming environments provide non-textual ele-

mentssee [31] for a survey. Proton [13] also introduced a

declarative syntax and interactive editor for describing se-

quences of events for touch-based devices. LiveWorld [30]

and several GUIbuilders allow users to set object properties

using property sheets, which list settable properties and

allow users to change them, sometimes updating the inter-

face to reflect their current values. These property sheets

can specify the look (colors, fonts, positions, etc.) of an

application but InterState incorporates states and constraints

to allow developers to also specify how an application be-

haves.


3/10

3

Event Languages and Models

Finally, many commercial and research systems have used

and augmented the event-action framework. Early event

models, like Sassafras [10] and the University of Alberta

User Interface Management System [7] inspired the fea-

tures of future commercial systems [18]. One interesting

extension of the standard event model is the elements,

events, & transitions model, which allows developers to

more concisely express how user interfaces should respondto user events [5]. Although such approaches have the po-

tential to make event-action driven user interfaces more

concise, they still split interactive behaviors code across

many locations.

DEVELOPING WITH INTERSTATE

InterState is implemented as a browser-based application.

Runtime and Editor

InterState separates its runtime (which maintains con-

straints, translates InterState objects into DOM objects in

the browser, and coordinates execution of the users pro-

gram) and editor. This allows developers to create applica-

tions using the editor and deploy them using only the

runtime. When the editor is open, it is displayed as a sepa-

rate window that communicates with the runtime window.

Because InterState is specifically tailored to implementing

GUI behavior, it is likely that programmers will want toimplement some non-UI portions of their applications out-

side of InterState. To support this, code running in Inter-

State can communicate with JavaScript in three ways: First,

InterState objects can call JavaScript functions on transi-

tions. Second, InterState objects can reference JavaScript

variables in exactly the same way they reference InterState

variables. Finally, JavaScript code can emit events to which

InterState objects react.Live Development

One of the important aspects of our development environ-

ment is that it is live changes made to the programs

source code are instantly reflected in the running applica-

tion. We decided that it was important to have a live devel-

opment environment for three reasons.

First, liveness can make the environment more approacha-

ble and beginner-friendly by helping to bridge the gulf of

evaluation [22], a significant barrier for new developers

[14]. Another important aspect of most live development

environments is that the developer always has a running

program. One great aspect of spreadsheet programming, for

instance, is that when the user makes a mistake in a particu-

lar cells formula, the entire spreadsheet does not stop

working [4,18]. Similarly, InterState allows errors to be

localized: cells with errors only prevent the parts of the

program from running that depend on those cells.

Second, liveness can enable the user to quickly evaluate the

design. Although syntactic errors can sometimes be made

immediately visible in edit-compile-run environments, live

programming allows both syntactic and semantic errors to

become immediately apparent by enabling developers to

immediately test their code. This is particularly important

because reflection-in-actionstepping back and evaluating

their design as designers are in the process of creating itis

a crucial part of the design process [29]. While sketches and

drawing applications allow designers to quickly evaluatethe lookof their application during the design process, In-

terState is designed to be one of the first tools to allow themto quickly evaluate thefeelof their application.

Finally, liveness can enable quick experimentation and pa-

rameter tweaking. Experimentation is a crucial part of the

design process and one that is not well supported by todays

development environments [8]. Again, it is relatively easy

to experiment with different application looks with sketch-

es, drawing programs, etc. However, it is more difficult to

change or experiment with the feel of the application.

INTERSTATE PRIMITIVES AND FEATURES

InterState expresses interactive behaviors by specifying

constraints that apply in particular states. There are three

basic entities in InterState: objects,properties, and cells. Inthis section, we will explain these three basic entities and

their features in detail.

Objects

Objectsmay contain any number of named fields1. A field

may be aproperty or another object. A field with a property

always has a value. Fields containing objects form a con-

tainment hierarchy, with a top-level object that is always

called sketch. Every object also has at least one associated

state machine, where all state machines start with a single

state by default. The InterState editor displays every ob-

jects state machine horizontally at the top of the object (see

Figure 1). As we will describe in more detail later, an object

may also inherit from other objects, in which case it willinherit the other objects fields and a copy of their state

machines and constraints. In the constraint and event ex-

pressions, the current object is called this.

Properties

A property maps every state and transition of an objects

state machine to a value. That value is either empty (repre-

sented as a grey circle) or a cell(represented as a rectangle

with the cells value). When a state is active or a transition

fires, the propertys value is computed from its value in thatstate or transition. If that value is empty, then the last value

remains in use. Cells set on states are evaluated as con-

straints, whereas cells set on transitionsare evaluated once,

to allow values to be stored. For example, in Figure 1, thex field is a property, and in the draggingstate, there is a

cell that constraints its value to the expression mouse.x. In

the transition from dragging to idle, its last x position is

stored. However, in the idle state, the property is empty,

so its last value continues to be used.

1 InterState does not distinguish between methods and fields, as Java

does. Instead, functions are first-class and can be used as a fields value.


4/10

4

Cells

Cells are meant to hold an individual value. That value can

be a constant (e.g. 50 or 'red') or it can be a constraint

(e.g. x+1or sqrt(y)) that recalculates its value based on a

formula (see Referencing below). InterState uses Con-

straintJS [24] as its underlying constraint solver, and sup-

ports one-way constraints with indirection (the target object

can be itself be calculated by a constraint), as in:this.object_at_left.color

In Figure 1, in the draggingstate, the xand yproperties

of rectare constrained to mouse.x and mouse.y respec-

tively.

State Machines

Behaviors often combine multiple state machines; an objectmight, for instance, be draggable and selectable. In order to

avoid the state explosionproblem where developers would

have to create combinatorial number of states (e.g.

dragging_and_selected, idle_and_selected, etc.),

InterState extends the standard finite-state machine model

in two ways. First, InterState allows state machines to be

concurrent, with multiple states active simultaneously

[9,12]. When multiple states are active, InterState uses left-

to-right precedence to choose which active value properties

should use, if there are conflicts. Second, it allows states tobe nested a state can hold any number of substates, al-

lowing some behaviors to be expressed more concisely [9].

In the current design of the interactive editor, states areadded by clicking the + button on the far right (see Figure

1). Transitions are added using a context menu that appears

using a right click. Every transition has an associated event,

which can be viewed and edited in the text box above the

transitions arrow. The trapezoidal shape of InterStates

states is designed to allocate a column every transition, hor-

izontally centered where the transitions arrow begins.

Transition Events

Most transition events are specified using the format:

on(, ), where

is a user event to listen for (e.g. 'mousedown') and

represents any number of parameters for the

event (e.g. the target object for mouse events or the delayfor timer events). Guardsmay be added to events to further

specify conditions when their transition should fire:

on('mousedown', this).when(this.is_active).

In addition to user events, developers often need to express

events referring to changes in fields valuesfor instance,

to express that a draggable object should be highlighted

when it overlaps with its drop target. In imperative lan-

guages, this would require that the developer change that

property through a setter, which would then emit a custom

event. InterState simplifies this by introducing constraint

eventsBoolean expressions like (this.x > 500)that

fire any time the value of the expression switches from

falseto true.

Higher-level Transition Events

Sometimes, we want to give semantic meaning to an event.

Thus, InterState includes a way to emit a higher-level event

when any transition fires by adding extra expressions after

the transition events expression. For example, the state

machine shown in Figure 3 can emit a triple-click event on

this object by setting the second transitions event to:

on('click', this); emit('tplclick', this)

These expressions can also call JavaScript functions:on('click', play_button);play(curr_song)

Inheritance

It is often useful to inherit properties. For example, suppose

we have an object that represents a square. Every square

should have properties for width and heightwhose val-

ues are equal. InterState enables such inheritance. Objects

inherit from another by specifying what objects they should

inherit from in their prototypesfield, as shown in Figure

4, where my_square inherits from square. Inherited

properties (e.g. my_square.height) are greyed out.

InterState inheritance is modeled after traditional prototype-

instance inheritance, but with several important differences.First, rather than inheriting property valuesInterState inher-

Figure 2.Syntax and runtime errors are highlighted in the edi-

tor but do not prevent the program from running.

Figure 4. Objects can inherit properties from other objects by

placing them in their prototypes field. Here, my_square in-

herits from square. Because my_squarehas defined a proper-

ty for widthbut not height, it uses its own definition of width

but inherits squares definition for height, as indicated by

the greyed out color. Note that my_squareinherits the defini-

tionof height, not the value. Thus, the variable widthevalu-

ates to a different value (20) than it does in square(15).

Figure 3.A state machine representing triple clicking.


5/10

5

its property definitions. This is illustrated in Figure 4, where

my_square inherits the definition of height, rather than

its value, since my_square.height should equal

my_square.width, not square.width. By inheriting the

definition rather than the value, InterState allows prototypesto define behaviors that use the state and property values of

the objects that inherit from them.

Second, unlike mostprototype-instance inheritance models,

InterState allows multiple inheritance; objects may inherit

from any number of other objects. This is expressed by en-

tering an array of objects into the prototypesfield. If an

InterState object inherits from multiple objects with con-

flicting values for the same field name, InterState uses left-

to-right precedence for conflict resolution. Object propertyvalues are combinedacross inherited values; if an objects

property is not defined for a state but it is in one of the ob-

jects prototypes, it inherits that prototypes definition for

the state. This allows multiple behaviors to control the sameproperty simultaneously. An example of this can be seen in

Figure 5, where my_selectable_draggable inherits

from both selectable and draggable. Because both

prototypes define color, the left-most value will be used;

if my_selectable_draggableis in the selectedstate,

it will be 'blue'; otherwise, it will be 'black'or 'red',

depending on the dragging state.

Third, when one InterState object inherits from another, it

also inherits a copyof that objects state machine. For ex-

ample, in Figure 5, my_selectable_draggable gets a

copy of the state machines for both selectable and

draggable. The fact that a copy of the state machine isinherited, rather than the state machine itself, is important;

we usually do not want all of the objects that inherit from a

particular object to be in the same state. For example, we do

not want every object that inherits from draggableto en-

ter the draggingstate when one of them does2.

2 If desired, having them all change state together could be implementedby changing target of the on('mousedown', this)event so that it is

We often want the same behavior to apply across multiple

unrelated objects. In order to enable this, other toolkits haveused the strategy of creating separate interactor objects that

describe behavior which are attached to graphical objects

[11,20]. However, InterStates standard notion of inher-itance includes behavior inheritance.

Finally, the prototypes field, like every other property,

can have different values in different states, and can even

be computed by constraints, allowing the prototypes of any

given object to depend on its current state and other fields.

For instance, an object might inherit from draggableonly

when it is enabled.

Field References & Scoping

To avoid naming conflicts, InterState scopes field name

references by object. Whenever a cell references a field

name as part of a constraint, InterState searches up its con-

tainment hierarchy to find that variable. If InterState cant

find a variable in the containment hierarchy, it then search-

es for JavaScript variables with the same name. This search

is done live, so that if properties change values or names,

then their reference also changes immediately. To illustrate,

suppose we have the following containment hierarchy:

x: 'far'

OBJ:

x: 'near'

prop_a: x

Then prop_a gets the value 'near'. If we delete the x

property of OBJ, then prop_achanges to 'far'. InterState

includes three reserved keywords:

this: the current object. parent: the current objects parent in the containment

hierarchy.

sketch: the top of the containment hierarchy.

something like on('mousedown', obj1). That way, every draggable

instance would use the object obj1as the event target.

Figure 5.An object that inherits from both draggableand selectableselectablebehaviors. Note that the values from the color

field are inherited from both draggable('red') and selectable('blue').


6/10

6

For instance, in the above snippet, if we wanted to specify

that OBJ.prop_arefers to the xproperty of OBJ, we could

use either this.xor OBJ.x.

Copies and Templating

Often, developers need a list of similar items to display and

do not want to declare a display for every object in that list,

either because it is too tedious or because that list of items

will be computed at runtime. In imperative languages, thisfunctionality has been implemented as list views in data-

binding libraries or maps (e.g. Amulet [20]) that allow pro-

grammers to specify a template display and to specify the

number of instances they want.

InterState handles this by adding an optional copies field

to ordinary objects. When copies is set to an array or a

number, its parent object then creates a set of items rather

than a single item. The editor visually signals this by dis-

playing a stack under the objects display (see Figure 6).

For every item, InterState sets two properties: my_copy,

which carries the value for a particular item (e.g. 'item1',

'item2') and copy_num, which carries the index for a

particular item (e.g. 0, 1). When the value of copies

changes (dynamically or by user edits), the list is updated

with respect to added, moved, and removed items instead of

recreating the entire list.

For example, suppose we have a color palette that shows a

tiny swatch for a set of colors that a user has set as favor-

ites. The list of favorite colors is stored in the favorites

variable as hex values (e.g. ['0x900', '0x333']). When

users click add favorite, a new element is pushed onto

that list and when they click remove favorite, the selected

element is removed. The developer wants to specify only

once how to display every swatch, by using the

color_dispprototype, and by expressing that a copy of itshould be created for every element in the favoriteslist.

They can set copiesto a constraint to favoritesand the

InterState runtime environment creates a copy of

color_disp for every element in the favorites array

(updated automatically). color_disp can then constrain

its fillproperty to be my_copy, so that every instance has

the appropriate color.

Note that the copies prototypesfields can be computed

by a constraint that depends on each copys my_copyfield.

For instance, in a directory viewer application, we could set

copies to the contents of the directory. Then, an item

could inherit from folder_viewif my_copyis a folder orfrom file_viewif my_copyis a file.

Queries

It is often useful to have a powerful way to express opera-tions on groups of objects to, for example, find every

checkbox that is in the selectedstate. InterState includes

a built-in function called find for making such queries

with a chaining syntax inspired by other query languages

and libraries like XQuery, jQuery, EET [5], and HANDS

[26]. For example, the aforementioned query could be rep-

resented with the expression:

find(chkboxes).in_state('selected')

Query expressions can be used in cells, for example, to

keep track of every selected checkbox, or they can be used

in transition events. For example, to create an event that

fires when no checkboxes are selected, we could write:find(chkboxes).in_state('selected').len()==0

Performance was a prime concern in our implementation of

queries. One way we optimized performance was by using a

chaining syntax that does not require re-evaluating expres-

sions further up the chain. Thus, in the above example, if

one chkboxes switches to or from the state 'selected',

we would not re-evaluate the expression find(chkboxes)

and constraints that depend on non-changed items in the list

returned by the query would not be re-evaluated.

Manipulating Visual Objects

Finally, all of these components must be connected to an

output mechanism. We have implemented two primary out-put mechanisms: one for creating HTMLDOM objects and

another for creating Scalable Vector Graphics (SVG) ob-

jects. We have also experimented with WebGLas an output

mechanism for creating 3D interfaces. We will limit our

discussion to SVGobjects, because three of the four exam-

ples in our Applications section use SVG objects (the

alternate InterState editor uses DOMoutput). The DOMand

WebGLoutput mechanisms are analogous.

To make an SVG object appear on screen, developers can

make that object inherit from one of seven types of SVG

objects, all children of the sketch.shapeobject: circle,

ellipse, image, rect, text,group, and path (which

includes lines, arcs and other custom shapes). All of theseprototypes declare default values for properties that control

how they are displayed (e.g. rect has a width attribute

and imagehas an srcattribute). Other attributes include:

animated_properties: an array of property namesthat should smoothly animate between values (if it is set

to true, every propertys changes will be animated)

Figure 6.An object (obj) with multiple copies. copiesis set to

['item1', 'item2', 'item3']. Every copy has two prop-

erties: my_copy, which is set to that copy's item and

copy_num, which is set to that copy's index. Here, we are look-

ing at the first copy (as indicated by the [0, length 3]).


7/10

7

animation_duration and animation_easing tocontrol the animations length and easing

showto determine if the object is visible or hiddenSometimes, we want to create SVGobjects that should not

be shown on screen. For example, we might create a

buttonprototype object that specifies how a button should

behave but we do not want buttonitself to appear. There-

fore, InterState only places objects in the runtime window if

they are children of the top-level screenobject.

USER STUDY

To evaluate the understandability of InterStates visual no-tation and the usability of its editor, we conducted a user

study. We recruited 20 experienced developers between the

ages of 19 and 41. Participants were mostly professional

developers or students and were geographically distributed

across the United States; half of our participants participat-

ed remotely using screen-sharing software.

Method

Participants were sequentially given two interactive behav-iors; one implemented in JavaScript using the RaphaelJS

drawing and event-handler library (called JS) and another

implemented with InterState (called IST). For each behav-

ior, participants were asked to make modifications to evalu-ate their ability to understand the implemented behavior and

express a new behavior.

For the first behavior (called B1) participants were given

code for a standard drag and drop behavior. Participants

were asked to implement drag lockan accessibility fea-

ture that allows users to double click an object to drag it

until they double clicked again. The second behavior

(called B2) was an image carousel that displayed a large

featured image and series of thumbnails that changed thefeatured image when clicked or auto-advanced automatical-

ly after a timeout. We asked participants to change displayfeatures of the thumbnails, the auto-advance interval, and to

add an indicator below the featured thumbnail to count

down with the auto-advance interval. The relative number

of lines of code and objects is detailed in Table 1.

Both of these behaviors were implemented in JavaScript

and in Interstate. Participants were given the same task de-

scription regardless of implementation. We counterbalanced

the order of the tasks, creating a total of four participant

groups (B1JS/B2IST; B1IST/B2JS; B2JS/B1IST; and B2IST/B1JS).

To make our comparison as fair as possible, we started with

third-party code for the JavaScript implementations and

simplified it by reducing boilerplate and adding descriptive

variable names that were consistent with those used in the

InterState implementations. We also used a live JavaS-cript editor (JSBin) that immediately re-evaluates JavaScript

snippets when their source changes. Finally, participantswere given tutorials in both JavaScript (with RaphaelJSand

JSBin) and InterState. They were also given comparable

reference sheets in for RaphaelJSJavaScript and InterState.

Results

Behavior 1 (Drag Lock)

Users were able to implement the drag lock task significant-

ly faster with InterState (two-tailed heteroscedastic Stu-dents t-test, p < 0.05). However, there was a large variance

in time needed between participants in the JavaScript condi-

tion. Although relatively few lines of code were required,

reasoning about callbacks timing proved challenging for

many users. Many participants used the strategy of consolelogging to help track their interfaces state.

Even after completing the task, many JavaScript partici-

pants were not confident in their implementations correct-

ness. As one participant in the B2IST/B1JS condition (who

completed the task relatively quickly) noted afterwards, I

dont know about all of the event combinations but I think I

got it right; I was more sure in InterState.

Behavior 2 (Image Carousel)

Behavior 2 was split into three sub-tasks. The first two sub-

tasks required users to find the variables that controlled the

thumbnail width and timing of auto-advances. Interestingly,

although participants were able to complete the overall task

significantly faster with the InterState editor (p < 0.05),

they were able to complete the variable modification sub-tasks significantly faster in JavaScript. This appeared to be

largely because of the different approaches required by edi-

tor. Whereas JavaScript participants were able to scan their

code for expected identifiers or search for certain keywords,

InterState users had to navigate to the relevant objectfirst.

The last sub-task required users to add a timer indicator.

Participants in both implementations used two strategies for

this: either creating an indicator for each thumbnail or cre-

ating one indicator that follows the featured thumbnail.

Figure 7.The relative times (in minutes) across 20 participants

to complete tasks in JavaScript (JS) and InterState (IST).

"## $%&' ()*+ "$# ,-'! .&%)/012

%&(3410 )5 .)61 78 9: ;? 9: ;@=

A .&22B&*+0 7 9: ;= < 9: ?=

'&(

A .1220 ;; 9:


8/10

8

Post-Survey and Interview

Most participants felt comfortable with InterStates visual

notation, calling it intuitive and clean. Nearly every

user cited InterStates ability to display the current applica-

tion state and property values live as its most useful fea-ture. This helped many users quickly debug and deduce the

meaning and roles of some properties. For example, in B2,

both implementations had a property that tracked the num-

ber of milliseconds before the featured image auto-advanced. Most JavaScript (B2JS) participants missed this

variable while most InterState participants found it, appar-

ently by observing how its value changed over time. Partic-

ipants also cited the ability to look at any object to find its

attributes as another useful feature of the InterState editor.

On the other hand, some participants expressed skepticism

that they would ever use a visual language in practice; evenone participant who agreed that InterState was a clear rep-

resentation of behaviors could not imagine programming

using any visual language. The most cited reason for this

was that participants still felt more comfortable with

standard imperative code. This may be largely due to their

relatively long exposure to standard code or the dearth ofcommercially available visual languages.

Discussion

Our user studies pointed to a number of ways to improve

future versions of the InterState editor. First, as a conse-

quence of how InterState lays out code, InterState develop-

ers often need to search to find objects. This is because

InterState requires property modifications to be written on

the propertys row. This is in contrast with JavaScript andmost imperative languages, where property modifications

can happen anywhere. This search could be textual or could

be in the form of an inspector that allows users to click on

objects in the runtime window to immediately open their

representation in the editor window (currently, InterState

only supports the opposite direction: highlighting runtime

objects when the user is editing their representation).

Additionally, when editing, participants often needed to

reference properties that were outside of the editors visual

scope, pointing to the need to increase the amount of infor-

mation available in the editor. We plan on exploring ways

to do this without increasing visual clutter.

One unexpected advantage of InterState over JavaScript

was that while performing JavaScript tasks, nearly every

participant broke (wrote code that did not compile) their

code at some point. This was largely because adding new

behavior to event-callback code often requires modifying

old callbacks in addition to adding new callbacks. By con-trast, adding new behavior to InterState examples largely

only required adding states, transitions, and cells.APPLICATIONS

To evaluate InterStates expressiveness, we implemented a

number of example applications. In this paper, we will de-

scribe four of these applications. Note that all of these ex-

amples were built entirely with InterState without any pre-

built widgets (as in no primitives for buttons, etc.).

Music Player & Playlist Editor

We implemented a music player and playlist editor shown

in Figure 8. This example illustrates how InterState can be

used to build front-ends that call code written in another

language; the actual audio output is controlled using JavaS-

cript functions that call the HTML5 audio API. It re-createsmany of the features of a desktop music player and playlist

manager, allowing users to add or remove songs fromplaylists, create new playlists, and to play songs and control

the volume and playback position.

This example also demonstrates how InterState can be used

to build complex real-world interfaces. It leverages Inter-

States use of constraints to make dealing with changing

model data (playlists and songs being added and removed)

trivial. It also uses behavior inheritance for its buttons and

sliders. Overall, this example was built using 11 InterState

objects, and 90 cells across 16 states with 21 transitions.

Breakout

We also implemented a version of the Breakout game. Inthis example, the position of the ball is updated by a transi-

tion that fires once per frame. Ball bounce events are im-

plemented as constraint events (when x


9/10

9

to level+1 in a transition). In this example, we increase

the speed and the number of rows every level. Overall, this

example required five InterState objects with 51 cells across

10 states and 21 transitions.

For comparison, we found an implmentation of breakout

written for the Web in JavaScript3. This example was

written by an expert as a tutorial for other developers and

was thus written as cleanly as possible with around 1,000lines of code. Despite being relatively cleanly written, it isdifficult to answer some basic questions about how the code

works just by looking at it. For example, if we wanted to

understand what might affect the balls position, we would

need to search across multiple (about seven in this case)

event handlers, which are in turn initialized in various non-

local parts of the code.

This kind of scatterring of functionality across multiple

locations in code is inherent to the event-callback style

required by most languages; either the code that affects a

particular property must be grouped (e.g., all of the event

listeners affecting the ball are part of the balls class) or allof the properties that an event affects must be grouped (e.g.,

everything that a keypress may affect is placed in one

keypress event handler).

Shape Editor

We implemented a shape editor that allows users to select

and manipulate properties of retained objects, including its

position, size (from all eight possible edges and corners),

rotation, and color. It also enables optional gridding and

keyboard modifiers to change the resize mode (e.g. resize

from center by holding CTRL). This example illustrates

InterStates ability to deal with applications that have a

large number of possible states; manipulable objects have

about 12 columns and its representation still fits on a single

15-inch laptop screen without scrolling.

InterState Editor

Finally, weve built an alternative InterState editor using its

own primitives. This editor displays InterState objects as atree. Its implementation looks at every InterState object and

decides which view to inherit from based on that objects

type (an object will inherit from object_view, a cell from

cell_view, etc). The object_view prototype has one

child_views property whose copies is set to that

objects list of properties. This allows the list of that

objects properties to change automatically with the

underlying data model. This example uses 76 cells across

34 objects and calls JavaScript code to manipulate theInterState objects. Although implemented as a proof-of-

concept, we are exploring the idea of writing future

versions of the InterState editor with its own primitives.

3github.com/jakesgordon/javascript-breakout/

IMPLEMENTATION

InterState is built in HTML and JavaScript using the Con-

straintJS constraint solver [24]. InterState also uses the es-

prima.org ECMAScript parser to generate constraints from

expressions written in cells. Communication between theInterState editor and runtime windows is done through a

wrapper layer using the HTMLchannel messaging API. The

InterState editor uses asynchronous constraints to track thevariable states and values in the runtime window. The edi-

tor sends edit commands to the runtime window through the

same wrapper layer. InterState objects can also be serialized

and use the HTMLlocal storage APIto save and load Inter-

State programs across sessions or as files.

FUTURE WORK

Syntax

The InterState constraint syntax is currently based on Ja-

vaScript expressions (enabling, but not requiring any of the

control structuresif, while, etc.). However, we plan on

investigating ways of making the syntax more beginner-

friendly, guided by commonalities in how users naturallydescribe behaviors [27].

Widgets and Modules

Widgets and example code can help beginners to get started

with any new language or development environment. Thus,

we want to create a set of white box widgetspre-built

components that can be used as-is, or whose source can be

modified or overridden, as necessary.

Other Features

Participants in our user study suggested a number of minor

improvements to our interface, including auto-complete,

automatically resizing state displays, and better debugging

support. We will investigate how to include these features.

CONCLUSION

In this paper, we presented the design and implementation

of InterState, a programming framework and developmentenvironment for creating interactive applications. We also

evaluated InterStates usability with a laboratory study and

its expressiveness with a series of example applications. We

plan on making the full source freely available in the near

future. We also plan on adding more features and creating

tutorials and documentation.

ACKNOWLEDGEMENTS

(Blank for anonymity)

REFERENCES

1. Appert, C. and Beaudouin-Lafon, M. SwingStates:

Adding state machines to Java and the Swing toolkit.

Software: Practice and Experience 38, 11 (2008), 1149

1182.


10/10

10

2. Blanch, R., Beaudouin-lafon, M., and Futurs, I.

Programming Rich Interactions using the Hierarchical

State Machine Toolkit.AVI, (2006), 5158.

3. Bostock, M., Ogievetsky, V., and Heer, J. D3: Data-

Driven Documents. Visualization and Computer

Graphics 17, 12 (2011), 23012309.

4. Burnett, M., Atwood, J., Djang, R.W., Gottfried, H.,

Reichwein, J., and Yang, S. Forms / 3!: A First-OrderVisual Language to Explore the Boundaries of the

Spreadsheet Paradigm.Functional Programming 11, 2(2001), 155206.

5. Frank, M.R. Model-Based User Interface Design By

Demonstration and By Interview. 1995.

6. Freeman-Benson, B. Kaleidoscope: Mixing Objects,Constraints, and Imperative Programming. OOPSLA,

(1990), 7788.

7. Green, M. A Survey of Three Dialogue Models.ACM

Transactions on Graphics 5, 3 (1987), 244275.

8. Grigoreanu, V., Fernandez, R., Inkpen, K., and

Robertson, G. What designers want: Needs of

interactive application designers. VL/HCC, (2009), 139

146.

9. Harel, D. Statecharts: A Visual Formalism for Complex

Systems. Science of Computer Programming 8, 3

(1987), 231274.

10. Hill, R.D. Supporting Concurrency, Communication,

and Synchronization in Human-Computer Sassafras

UIMS.ACM Transactions on Graphics 5, 3 (1987),

179210.

11. Hudson, S.E. and Mankoff, J. Extensible Input Handling

in the subArctic Toolkit. CHI, (2005), 381390.

12. Jacob, R.J.K. A State Transition Diagram Language for

Visual Programming. Computer 18, 8 (1985), 5159.

13. Kin, K., Hartmann, B., DeRose, T., and Agrawala, M.

Proton: Multitouch Gestures as Regular Expressions.

CHI, (2012), 28852894.

14. Ko, A.J., Myers, B.A., and Aung, H.H. Six Learning

Barriers in End-User Programming. VL/HCC, (2004),

199206.

15. Lecoanet, P., Lemort, A., Mertz, C., et al. Revisiting

Visual Interface Programming: Creating GUI Tools for

Designers and Programmers. UIST, (2004), 267276.

16. Letondal, C., Chatty, S., Phillips, W.G., and Andr, F.Usability requirements for interaction-orienteddevelopment tools.PPIG, (2010), 1226.

17. Meyerovich, L., Guha, A., and Baskin, J. Flapjax: A

Programming Language for Ajax Applications.

OOPSLA, (2009), 120.

18. Myers, B., Hudson, S.E., and Pausch, R. Past, Present,

and Future of User Interface Software Tools. TOCHI 7,

1 (2000), 328.

19. Myers, B., Park, S.Y., Nakano, Y., Mueller, G., and Ko,

A. How Designers Design and Program InteractiveBehaviors. VL/HCC, (2008), 177184.

20. Myers, B.A., Mcdaniel, R., Miller, R., et al. The Amulet

Environment!: New Models for Effective User InterfaceSoftware Development.IEEE Transactions on Software

Engineering 23, 6 (1997), 347365.

21. Myers, B.A. Separating Application Code fromToolkits: Eliminating the Spaghetti of Callbacks. UIST,

(1991), 211220.

22. Norman, D. The Design of Everyday Things. Doubleday,

New York, New York, USA, 1988.

23. Olsen, D.R. User Interface Management Systems:

Models and Algorithms. Morgan Kaufmann, San Mateo,

CA, 1992.

24. Oney, S., Myers, B., and Brandt, J. ConstraintJS:

Programming Interactive Behaviors for the Web by

Integrating Constraints and States. UIST, (2012), 229

238.

25. Ozenc, F.K., Kim, M., Zimmerman, J., Oney, S., and

Myers, B. How to support designers in getting hold of

the immaterial material of software. CHI, ACM Press

(2010), 25132522.

26. Pane, J.F., Myers, B.A., and Miller, L.B. Using HCI

Techniques to Design a More Usable Programming

System.HCC, (2002), 198.

27. Park, S.Y., Myers, B., and Ko, A.J. Designers Natural

Descriptions of Interactive Behaviors. VL/HCC, (2008),

185188.

28. Samek, M. Who Moved My State?Dr. Dobbs Journal,

2003.

29. Schn. The Reflective Practitioner. Temple Smith,

London, England, 1983.

30. Travers, M. Recursive Interfaces for Reactive Objects.

CHI, (1986), 379385.

31. Zhang, K. Visual Languages and Applications. Springer,

2007.

about interestate

Documents

Transcript of about interestate