Tangible Interfaces

Readings

• Hiroshi Ishii, Brygg Ullmer, “Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms”, Proceedings of CHI ’97, March 1997, pp. 234-241.http://doi.acm.org/10.1145/258549.258715 Better figures here: http://www.acm.org/sigchi/chi97/proceedings/paper/hi.htm

• Roy Want, Kenneth Fishkin, Anuj Gujar, Beverly Harrison, “Bridging Physical and Virtual Worlds with Electronic Tags”, Proceedings of CHI ’99, May 1999, pp. 370-377.http://doi.acm.org/10.1145/302979.303111

More Readings

• Jun Rekimoto, Brygg Ullmer, Haruo Oba, “DataTiles: A Modular for Mixed Physical and Graphical Interactions”, Proceedings of CHI ’01, March 2001, pp. 269-276.http://doi.acm.org/10.1145/365024.365115

• Jun Rekimoto, Eduardo Sciammarella, “ToolStone: Effective Use of the Physical Manipulation Vocabularies of Input Devices”, Proceedings of UIST ’00, Nov. 2000, pp. 109-117.http://doi.acm.org/10.1145/354401.354421

Supplemental Readings

• John Underkoffler, Hiroshi Ishii, “Urp: A Luminous-Tangible Workbench for Urban Planning and Design”, Proceedings of CHI ’99, May 1999, pp. 386-393.http://doi.acm.org/10.1145/302979.303114

• Leo Tolstoy, War and Peace, 1869, English translation by Rosemary Edmonds, Viking Press, 1982. http://www.online-literature.com/tolstoy/war_and_peace/

Motivation

• Desire to remove divide between electronic and physical worlds– Bring benefits of computation (the virtual world)

beyond confines of the screen, to the rest of life• Ubiquitous computing (Weiser)• Augmented reality

– Making crossing between “bits and atoms” seamless (or at least a lot easier)

• Put information “in our hands” in a rather literal way

Motivation (cont)

• “Spatial thinking”– Concrete– Like sketching: fluid, natural, dexterous actions– (Graspable) physical objects provide strong

affordances (much closer to real affordances instead of virtual)

– Different sets of thinking / reasoning skills• Used to say “visual specification for visual things”

– E.g., UI layout“spatial specification for spatial things”

In “Tangible Bits” paperIshii Promotes Three Concepts• Interactive surfaces

– Transform (all?) surfaces into an interface• (Really nothing special about “surfaces” except that they are

convenient to work with at this stage)

• Coupling of bits to atoms– Couple information associated with (relevant to)

physical objects to the objects• Provide direct access to info from object• Right information is always “at hand”

• Ambient media– Ability to use the periphery of attention and move

between the periphery and center of attention

Prior Work Examples (From Ishii’s “Tangible Bits”)

• DigitalDesk– Wellner, EuroParc

– Desk with top-projected video– Camera tracking from above– Digitizing tablet– Microphone

• On surface

DigitalDesk (cont)

• Interact with real paper– Digitize with camera

(eventually) and recognize action gestures

– Augmented with video from above

– Used microphone to detect “taps”

• Difficult to detect contact via video alone

Prior Work Examples (From Ishii’s “Tangible Bits”)

• Marble Answering Machine– Durrell Bishop, RCA

– Marbles represent calls– Put marble in indentation to

play message– Another spot to call back

• Live Wire – Natalie Jeremijenko, Xerox PARC

– String turned slightly on each Ethernet packet

– Making invisible bits visible/tangible

Ishii’s work in “Tangible Bits” paper

• metaDesk– Rear projected video (the bits)– Physical objects tracked on top (the atoms)

• Manipulating physical objs affects the bits (not rev)

metaDesk

Applications of MetaDesk

• Tangible Geospace– Map manipulation via phicons, etc.

• Two phicons (representing particular places) allow very easy manipulations of position, orientation, and scale of the map

– Solid Plexiglas shape

• Active lens gives 3D views– LCD display on a tracked arm

• Passive lens gives augmented or filtered 2d views– Clear lens – Change the projected display under the lens!

MetaDesk Issues

• Good demo, but not a real application– Solid and compelling apps for tangible interaction still

somewhat of an issue, overall– Seems very compelling, but is their really a big benefit

• Seems somewhat domain specific (see URP & ToonTown later)

• Do I have to have a new physical object for every command/parameter/whatnot?– Loosing some of the extreme flexibility of the virtual

world– Big research issues: How do we find the right mix of

physical / virtual to get good properties of both?

AmbientRoom

• Video

AmbientRoom

• Interesting things here:– Ambient displays

• Aesthetics important• A start on user defined mappings

– Does this scale?

– What about all the “hidden state” (can’t see what you’ve spec’ed)?

– Physical containers for information

• Issues– Hamster awareness !?!

• Again, missing a compelling application

TransBoard

• Augmented whiteboard– Tracked pens, front projected image– Looking at “surfaces which absorb information

from the physical world”

• Collaborage we saw earlier (but done later) was similar but probably more compelling

Additional Apps for MetaDesk

• Illuminating Light– Optical bench simulation– “I/O bulb” concept– New color bar tracking (from above)

Video…

Additional Apps for MetaDesk

• URP – Urban planning app

– Physical objects cast virtual shadows and reflections

– Wind flow simulation– Tools for making

measurements

Finding a good mix of physical and virtual: DataTiles (Rekimoto)

• LCD with pen input– Transparent tiles

• May have preprinted information

• May have groves affording particular interactions

– Sit on pedestals • Serve as ID readers for tiles

DataTiles

• Appearance is combination of display and pre-printed

• Grooves afford particular actions (labeled as to purpose)

Interesting tiles

• Containers– Copy virtual objects, settings, etc. to a tile– Set aside until later

• Parameters– Search times with parameters– Dynamic query on (preprinted) map tile

• Grooves for sliders

Interesting tiles (cont)

• Actions– Print

• Generic print tile• Set next to object, join with gesture, fill in print dialog

– Time machine• Show previous states of another tile• “Clock” grooves with appropriate dynamic feedback

– Remote interaction tiles• Shared drawing surface with remote participants

Physical / virtual or generic / concrete balance

• Note that a range of interaction techniques are possible along scale of very concrete and pre-determined to very generic – Also static to dynamic and physical to virtual

• Composition mechanism allows fairly pre-determined tiles to have much more utility– General lesson: simple actions + composition

• Each small and simple

– Let combinatorics take over to give wide expressive range, but with (potentially) less overall complexity

Generally useful technology in this domain: Tagging objects

Tagging Objects

• Several technologies (see below) for tagging objects to enable sensing of identity– Objects brought into proximity of special reader

• Range varies: room to a few cm

– Reader determines a unique identifier for object• Typically a unique number• Note that 64 bits is probably enough to uniquely tag

every individual object ever manufactured– every CD, ever can of soup, every page of paper – Even 32 bits goes a long way (4 billion unique IDs)– 96-128 bits is better (can partition ID space conveniently)

Fun with IDs

• A number of things you can do with an ID– If you have network connectivity (or can cache) at the

reader then any identifiable object can have an arbitrary amount of data attached to it

• Look up the ID in a database to retrieve “attached data”• Data can be general info, commands, parameters, anything, …

– Attached object can be static or dynamic

• Can also do this with a local database if info is local

– Object can now be a surrogate for the data• Arbitrary data (incl. arbitrary programs) very powerful

Identified Objects Can Act as• Container

– Can (appear to) put in, carry away, and (later/elsewhere) take out (arbitrary) data

• User ID– Can serve to represent / identify a person

• Context ID– Can serve to identify a place – Can indicate the presence of a piece of equipment

• Provide handle for accessing it• E.g., walk into a room with a printer and get access to its queue

• Actions to be performed• Set of attributes / parameters / settings

ID Technologies

• Printed Bar Codes– Read by optical scanner

• Laser or other• Usually close contact, but can be done at a distance• Typical read error rates: 1 per 10,000

– Most common: UPC on all retail items

• Most common: 10 decimal digits + prefix & check digits

– Enough for every kind of soup, but not every individual can

Bar Codes (cont)

• Can also store a lot more data– See: PA vehicle registration

UPS “maxicode”

• Advantages of bar codes– Very cheap (just print them with a printer)– No battery needed for tag– Reliable, readers relatively cheap

• Disadvantages– Requires contact (and/or careful alignment)– Only work on relatively flat/smooth surfaces– Ugly

Aside: Can get rid of Ugliness

• Xerox DataGlyph technology designed to look better – Appears as a textured gray region

• Relatively high density• Also survives bad faxes and other distortions well

IR Beacons

• Encode ID in pulses of (invisible) infrared light – Same technology as TV remotes (cheap & easy)

– ID sent periodically (typically every few seconds)

• Applications– Fixed beacons to indicate place

• Example: HP CoolTown project– Emits URL of web page giving access to services (e.g., print

queue for printer, or control of projector) in room

– Active Badges identify location of a person

Other Interesting ID Mechanisms

• Contact tags, magnetic stripe, and smart cards• For limited sets of objects: weight them

– 0.1 gram resolution scales distinguish most objects– Issues with “wear and tear”

• May want to augment with vision

• Fingerprints– User fingertips as data repositories (!)

• Virtually “pick up” arbitrary data, carry, and “drop” it.

– Another potential app: 10 actions for 1 button

RFID Tags

• Probably most generally useful technology

• Reader has a typically large coil– Emits RF at a particular frequency

• Tag has a typically small coil– Inductively couples with reader– Draws power from it (no batteries in tag)– Modulates its load on inductive coupling to send

information back to the reader

A variety of form factors for tags

Different tag properties

• Embedded processors• Persistent memory• Single read vs. multiple

– Multiple requires special protocol with reader– Implies a processor

• Differing read ranges– Typical 10cm– With “active” tags (battery) can get longer (10m)

Advantages (from Want paper)

• Robustness– Designed to work in harsh environments

• Inside dogs, on pigs ears, nailed into wood,under labels at K-Mart

• No maintenance

• Post-hoc augmentation• Can typically be hidden (no aesthetic costs)

– Small and can be embedded in lots of things

• Easily sensed– Doesn’t require direct contact, line of sight, or alignment

Disadvantages (from Want paper)

• Need interface for associating meaning with objects– Not particular to RFID tags

• Knowing what is tagged– Hidden tags mean no affordance

A good application using object identification

• ToonTown access control interface in the SomeWire audio space Andrew Singer, Debby Hindus, Lisa Stifelman, and

Sean White, “Tangible Progress: Less Is More In Somewire Audio Spaces”, Proceedings of CHI ’99, May 1999, pp. 104-111, 625. http://doi.acm.org/10.1145/302979.303007

ToonTown

• Shared audio system with reciprocity– Needs good access control

• “Who can hear me” (awareness and easy control)

– Also supported localization of the audio• Can place participants left, right, etc.

• Place “cartoon characters” (small dolls) on trays to represent people– RF Tag in the bottom of each doll, readers in tray

ToonTown

• On tray can hear and be heard• Clear visibility, easy to manipulate• Downside: doesn’t scale (but not clear what would)

ToonTown

• Location on tray also controls apparent sound source location and volume

• Note spots at right for other UI functions

Toolstones

• An input device– Not exclusively tangible (used with GUIs), but a

lot to do with grasp and touch– Device designed for use by non-dominant hand

• Typically with a pen or other pointer in the dominant hand

– Note: In most GUIs if you aren’t typing you might as well sit on your other hand

• May be intermittent – Adjust, work with primary hand, adjust, etc.

About bimanual operations (cooperative between hands)

• Hands tend to take on different roles – A lot more than a “hand preference” or

“stronger hand”– Semantically different actions by dominant and

non-dominant hands

• Seminal work Yves Guiard, “Asymmetric Division of Labor in Human

Skilled Bimanual Action: The Kinematic Chain as a Model”, Journal of Motor Behavior, Vol. 19, No. 4, 1987, pp. 486-517.http://cogprints.soton.ac.uk/documents/disk0/00/00/06/25/

Three key observations from Guiard

(Assuming right-handed -- tendencies somewhat less clear left-handers)

• Left hand sets context for right– Left provides overall positioning and orientation

for detail work of right– E.g., left holds fabric, right stitches where fabric

held

Three key observations from Guiard (cont)

• Tendency for larger scale and lower frequency movements by the left hand– Larger, courser, slower motions by left– Smaller, more precise, faster motions by right

• Motion by left hand tend to precede those of the right– Position objects before acting on them– Pick up the cloth before sewing it

Working with these tendencies

• Taking proper advantage of different roles (rather than just subdividing the work) can provide better/smoother performance

• ToolStone device designed for non-dominant hand– Will see for tasks like selecting color from

palette instead of tasks like painting

ToolStones

• Rectangular device – Each of 6 faces – Distinguishable by touch alone (via size and/or marker)

• Works with a digitizing tablet– Uses components from particular type of tablet– Provides constrained six degree of freedom

(6 DOF) sensing• X,Y,Z position• 3 degrees of orientation (e.g., tilt, yaw roll)• constrained because stone must stay in contact with the

surface – typically one face must be on surface, but version supporting tilting

on edges also possible

Sensing

• Quantize possible positions and orientations out of full 6DOF space

• 8 Directions

• On 6 different faces

• Also track x,y position on tablet

Sensing

• Implemented using guts from 3 tablet pens

Sensing

• One “pen” in contact (close proximity) with surface for any orientation

• Pens are passive– No battery, just resonant coil sensed from tablet

• Each pen allows sensing of– Contact (very close proximity, not “tip switch”)– Angles with respect to tablet– “Pressure” and ID of pen

• Actually location of an iron slug within the coil• Each unit initially tuned to slightly different frequency• Moving iron slug changes within a range• Tablet can sense at least 3 different pens at simultaneously

Alternate sensing design

• Two pens in contact for any face

• Can sense tilt along edge

Interaction techniques

• Typically each face has a different task – Important that they be distinguishable by feel– But labeling also important / useful

• Palettes / menus– Orientation based menus (8 items max)

• Fitts’ law says…?

– Larger use motion to pick like a mouse• E.g. color palette

• Potential issue: What is it?…

…l…

Interaction techniques

• Typically each face has a different task – Important that they be distinguishable by feel– But labeling also important / useful

• Palettes / menus– Orientation based menus (8 items max)

• Fitts’ law says…?

– Larger use motion to pick like a mouse• E.g. color palette

• Potential issue: don’t have a button– Best for modal interactions such at tool palette or as

toolglass style interaction (will see this in a later lecture)– Basically piggyback off “primary” locator in dom hand

Interaction Techniques

• Also can do “Action + 1 scalar parameter”– E.g., Scale operation and magnitude of scaling– Action set by orientation

• Twist to get the one you want

– Parameter set by movement “up & down” (i.e, in direction of action selection)

• Absolute or clutched like mouse

• Could also do “Action + 2” but starting to get away from natural tendencies of non-dominant hand

• Can also do things like 3D object or camera or positioning that require more 2+ DOF

Other forms

• Doesn’t have to be rectangular or even have parallel sides– Although lots of flat surfaces seems

useful

• More tactile differentiation of faces might be useful

• Could also augment with buttons– But could easily get beyond spirit of

device for non-dominant hand