Using the Crowd to Understand and Adapt User Interfaces

Jeffrey NicholsIBM Research – Almadenjwnichols@us.ibm.com

Engineering!

Benefits of Model-Based Engineering

Develop for multiple devices simultaneously (e.g., web & mobile)

Create personalized interfaces• Accessibility (physical, cognitive, etc.)

• Previous experience

Compose interfaces from different components never before used together

And many more…

“... the main shortcoming of the model-based approach is that building models is difficult. A rich model describing all the features of an interface is a complex object, and hence non-trivial to specify.”

Puerta and Szekely, CHI’1994quoted in Meixner, Paterno, Vanderdonckt. i-com 2011

Why is building models difficult?

Requires humans (mostly)• If no interface exists, must investigate existing practice to determine

tasks, etc.• Interpreting and understanding human behavior is hard for computers• Automation may be possible if an interface already exists

E.g., [Gimblett & Thimbleby, EICS 2010]

Requires experts• Knowledge of abstract concepts and formalisms• Knowledge of specific languages and conventions• Tools may decrease the amount of expertise needed

QuestionsIf we use a crowd…

• Do we need models?

• Do we need experts to create models?

Outline

Brief overview of relevant crowdsourcing research

Examples using the Crowd to adapt & understand interfaces• CoScripter• Highlight & CoCo• Crowd-Created Models

Conclusions

CrowdsourcingA Brief Overviewadapted from materials by Michael Bernstein and others

Origin of the term

“Taking [...] a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call.”

Jeff HoweWIRED, 2006

Is this concept new?• In 1760, the British Nautical

Almanac distributed work of creating navigational charts through postal mail.

• Work was computed by two independent workers and verified by a third

• Process repeated since for other large calculations, e.g. Mathematical Tables Project

Recent Examples…

Paid Crowdsourcing

Pay small amounts of money for short tasks

Example Site: Amazon Mechanical Turk

• Roughly five million tasks completed per year at 1-5¢ each [Ipeirotis 2010]

• Population: 40% U.S., 40% India, 20% elsewhere

• Gender, education and income are close mirrors of overall population distributions [Ross 2010]

Major Topics of ResearchCrowd algorithms [Little et al., HCOMP 2009]

Incentives and Quality [Mason and Watts, HCOMP 2009][Dow et al., CSCW 2012]

Crowd-powered systems [Bernstein et al., UIST 2010][Bigham et al., UIST 2010]

AI for HCOMP [Dai, Mausam & Weld, AAAI 2010]

Complex Work [Kittur et al., UIST 2011]

Major Topics of ResearchCrowd algorithms [Little et al., HCOMP 2009]

Incentives and Quality [Mason and Watts, HCOMP 2009][Dow et al., CSCW 2012]

Crowd-powered systems [Bernstein et al., UIST 2010][Bigham et al., UIST 2010]

AI for HCOMP [Dai, Mausam & Weld, AAAI 2010]

Complex Work [Kittur et al., UIST 2011]

Crowdsourcing Algorithms

Essentially a workflow where each step may be performed by a different worker

Iterative algorithms [Little et al. 2009]

Digital Assembly LineCrowdFlower

Crowd-Powered Systems

Embed crowd intelligence inside of user interfaces and applications we use today

User Interface Wizard of Oz

Crowd-Powered Systems

Embed crowd intelligence inside of user interfaces and applications we use today

Wizard of Crowd

http://www.youtube.com/watch?v=n_miZqsPwsc

Real-time CrowdsourcingUsing recent techniques, it is now possible to harness crowd workers to solve tasks in near real-time[Bernstein et al. UIST ’11, Lasecki et al. UIST ’11 and UIST ’12]

Example: Real-time captioning using shotgun gene sequencing techniques [Lasecki et al. UIST ’12]

Legion [Lasecki et al. UIST ’11]

http://www.youtube.com/watch?v=P_Tqn-3BF_I

Crowdsourcingwrap-up

There is a lot of power available in the crowd…

How can we harness it to help engineer new or improved interfaces?

CoScripterA Wikipedia-style approach to crowdsourcing task traces

CoScripter: Capture & Reuse Web Tasks

• Employees in large enterprisesneed to share “how-to” knowledge

• This knowledge is typically keptby a few knowledge hubs

• The CoScripter approach:• Capture web tasks by watching

people do them• Automate repetitive tasks to save time• Use a natural-language scripting

language for understandability

[Leshed et al, CoScripter: Automating & SharingHow-To Knowledge in the Enterprise, CHI 2008]

http://coscripter.almaden.ibm.com/coscripter/browse/video

CoScripter Key Features

• Browser extension for recording and playback

• Wiki for storing, sharing, and collaboratively improving scripts

• Personal database allows use of sensitive information within scripts without sharing that information

30

CoScripter AdoptionDeployed inside IBM since Oct 2006– ~4200 users, ~3500 scripts

Deployed on public internet 2007-2012– ~13300 users, ~16,000 scripts

Interviews and analysis showsit addresses pain points– IBMers use it to automate

repetitive tasks and shareprocess knowledge with eachother

Time

# sc

ripts

Conclusions

• CoScripter provides an inexpensive and lightweight method to adapt existing interfaces

• Allows a crowd of users to produce a knowledge base of important interaction traces

• Resulting knowledge base could be used for many purposes• Re-authoring user interfaces for

different user groups• Generating models?

Highlight & CoCoadapting existing user interfaces to different contexts

Example: AA.com – Flight Tracking

AA.com Flight Tracking: Mobilized

Highlight, Nichols et al. IUI 2008, UIST 2008

AA.com Flight Tracking: Speechified

User: “What is the status for my American Airlines flight?”

System: “What is the flight number?”

User: “144”

System: “Flight Status – Arrived”

CoCo, Lau et al. UIST 2010

Highlight

Goals

Allow end users to create their own mobile “applications” for particular tasks

• No programming required

• Possible for any existing site

• All design decisions made by users

Allow programmers to extend capabilities of mobile applications

mobile user

proxy server

proxy browserweb server

webserver

user

mobile app designer(browser extension)

Highlight Architecture

How do end users create applications?

Highlight Designer• Built using Firefox web

browser

• Allows user to demonstrate a “trace” of interaction

• Direct manipulation tools

• Generalization allows creation of mobile apps with complex structure

Nichols & Lau, IUI 2008

http://www.youtube.com/watch?v=1H1wIIbXtec

How does it work?:Remote Control Metaphor

144

144

Remote Control Metaphor Discussion

Benefits• No need to understand

underlying code or describe application with complex models

• Working at the interactive level lets authors work with what they can “see.”

• Possible for end users, extensible by programmers

• If easy enough, allows users to create user interfaces that reflect their own needs and abilities

Drawbacks• Always running original interface

in the background

• Constrained by original design

• How to communicate those constraints to the author?

CoCo

The CoCo research vision

Explore the use of conversational user interfaces to web tasks

Design and build intelligent agents that:• Interact with the web on a user's behalf

• Converse with the user to clarify meaning

• Learn new knowledge over time

• Are personalized for a user's needs

Goal: improve user productivity and increase access to information technology through simpler interfaces

Not shown in the talk, but instructive (ACM DL access required):

http://dl.acm.org/ft_gateway.cfm?id=1866067&type=swf

Alice: punch out 17 30CoCo: Extracted this script from your logs:

Go to timecard.com/cocompany, enter yourpassword into the textbox, click Go...Run it?

Alice: yesCoCo: I don't know what “password” to useAlice: punch out 17 30 using alice00 as passwordCoCo: I will run your script using params

password=alice00CoCo: 17:30 Exit

scripts

CoScripter

CRH(browser logs)

Task knowledge Web automation

...Highlight

clips

Two paths to determining process:

Automatic• System finds existing script in database or infers script

from web history• Content is clipped based on heuristics matching original

command

Manual (“re-authoring”)• User creates a script in CoScripter• Specifies parameters as “personal database” values• Specifies “clip” commands to return information

Conclusions

• Relatively simple understanding is used to facilitate substantial changes to the UI

• CoCo leverages crowd-generated scripts. Highlight could have leveraged crowd similar to CoScripter

• Models are used to facilitate re-authoring, but not using a typical approach

• More robust underlying models could lead to more robust results

Crowd-Created ModelsAn initial exploration of using novice crowd workers to produce abstract models

Background

• Our initial re-authoring work relied on interactions being the same in each platform (Highlight), or a priori known transformations (CoCo)

• To make deeper changes, we knew that we would need deeper models

• Could the crowd help us build the models we need?

Disclaimer: This is work is initial and incomplete. If you would like to continue it, please let me know!

Process

1. Build domain model• What are the objects that are viewed

and manipulated?• What functions and parameters do they

have?

2. Build “task” model linked to the domain model• Primarily based on object functions

3. Collect traces for carrying out tasks• Integrated into Mturk the CoScripter

variant PlayByPlay [Wiltse et al. CHI 2010]

Building a Domain Model

Task• Give a link to web UI• Ask a question• Possibly use taboo words as in ESP

game

Goal• Questions designed to elicit nouns or

verbs that would correspond to object or function names

• Previously collected terms used in questions

• Iterate between noun & verb questions

Q: What can you search for?taboo: restaurants, hairdressers

Q: What things can you manipulate?taboo: restaurants, hairdressers

Q: What can you do with a restaurant?taboo: make reservation, see reviews

Q: What can you make a reservation for?taboo: restaurants, hairdressers

Results

Crowd workers don’t know abstract terms• E.g., What is an object? What are things you can

manipulate?• Phrasing in terms of concrete activities helps

• What can you search for?• What can you do with?

Lack of a clear widget model on the web makes interpreting demonstrations hard• When is a user using a custom component vs. a standard

one?

Take-aways

• With some cleverness, it should be possible to use novice crowd workers to construct useful models

• It will be a significant undertaking; probably at the level of a Ph.D. thesis

• Interns do not have time to complete a second thesis during the summer (even good ones!)

Conclusions…

Harnessing the crowd offers tremendous potential for engineering interactive systems…

…to provide new functionality

…to adapt UIs to specific use cases or platforms

…to understand UIs

Crowd Papers at EICS 2013

CrowdStudy: Extensible Toolkit for Crowdsourced Evaluation of Web InterfacesMichael Nebeling, Maximilian Speicher, Moira Norrie

CrowdAdapt: Enabling Crowdsourced Adaptation of Web Sites for Individual Viewing Conditions and PreferencesMichael Nebeling, Maximilian Speicher, Moira Norrie

Echo: The Editor’s Wisdom with the Elegance of a MagazineJoshua Hailpern, Bernardo Huberman

Crowdsourcing User Interface Adaptations for Minimizing the Bloat in Enterprise Applications (poster)Pierre Akiki, Arosha Bandara, Yijun Yu

How can you use the crowd in your research?

Thanks!

For more information, contact:Jeffrey Nicholsjwnichols@us.ibm.com

Crowd Resources

• WWW 2011 Tutorial by Panos Ipeirotis & Praveen Paritoshhttp://www.behind-the-enemy-lines.com/2011/04/tutorial-on-crowdsourcing-and-human.html

• Michael Bernstein CS 276 Lecturehttp://hci.stanford.edu/courses/cs376/2013/lectures/2013-05-08-crowdsourcing/CS376-2013-crowdsourcing.pdf

• Crowd Research bloghttp://crowdresearch.org/blog/