Designing for Touch... and Beyond - Josh Clark

josh clark@globalmoxiehttp://bit.ly/touch-wkshop

designing forTOUCH

SCIENCE!

Friends, that’s what we’re here for.To study the storied science of web design.

Part of the process is realizing, every so often,that we’ve been doing SOMETHING wrong all along.

That’s the nature of science and discovery:Periodic realizations that our assumptions need a tweak.A flash of insight that we’ve been understanding things naively.

10 years ago: web standards.We have to separate content from presentation.

3 years ago: responsive design.The web isn’t about one screen size.

THERE IS NO ONE TRUE OUTPUT FOR THE WEB.

There’s a new shift brewing. And it’s this realization.

There is noone true inputfor the web.

There is no one true INPUT for the web.

In other words: the web is not just for the mouse.

Responsive: we have to anticipate a variety of displays.New wrinkle: we have to anticipate a variety of inputs, too.

Like most truths, this one’s been staring us in the face all along.The web has always been device agnostic.

It embraces:text-only devices.Screen readers.Speech is coming on strong,Natural gesture, too.

LOTS of different inputs coming our way.

But the big input-related design transition that we’re confronting now...and really we started confronting five years ago with mobile...

is touch.

Because right now, hands and fingers are coming to life on our interfaces...

no, it’s okay.

TOUCHTHAT

As advertised, this talk is about designing for touch.

We’ve been dealing with touch on smartphones, and tablets are coming on strong.

So sure, I’ll talk about guidelines for designing on phonesand tablets. But perhaps more important, I’ll also talk aboutdesigning for touch on the desktop.

[next]Because friends, touch has erupted on the desktop,in just the last few monthsAnd that has some thorny implications for responsive design.

ULTIMATELY, IT MEANS THIS:All desktop web designs have to be touch-friendly.[repeat]

But first...let’s begin at the beginning:With the basics, with first principles.

[twitter]31% of US adults own a tablet. Up from 12% a year ago, and basically zero 3 yrs ago. http://j.mp/Xgyspe[/twitter]

Jan 2013:http://pewinternet.org/Commentary/2012/February/Pew-Internet-Mobile.aspx

PIXELS ARE THE BUILDING BLOCK

I have an ugly truth for you, friends.

To date, we’ve thought about web design as a visual pursuit.

A feat of visual design, of information design.

So we naturally approach our work in visual terms,we think in pixels.

But when you add touch to the mix...[next]we go beyond the visual... and to the physical.

Easy big guy. Easy.

It’s not only how your pixels

LOOKbut how they

FEELIt’s not just how your pixels look.But how they feel.How do your pixels feel in the hand?

The very nature of our job is changing.

[twitter]Designing for touch means: it’s not just how your pixels LOOK… but how they FEEL in the hand.[/twitter]

where is

TOUCH?

We’ve been dealing with phones

In fact, 31% of US adults own a tablet now.THIRTY ONE! A THIRD!Was only 12% a year ago.Basically zero three years ago.

The iPad is the fastest growing consumer productin the history of consumer products. Amazing.Touch is already a huge mainstream input.

So before diving into details of touch design,want to talk about how these tablets are used.And who uses them.

We call this mobile. But really it’s portable.Use the phone on the way to the coffee shop,but we use this AT the coffee shop.Almost always used in settled environments.

iPad has very different mindsets because it’s in much more settled environments.

10%of US adults own a tablet

DECEMBER 2011

iPad came out in March 2010.

At the beginning of december in 2011.but look what happened the next month.

And here’s where we stand as of January 2014.That’s up 8% from 34% in September 2013.

For households making over $75k USD, it’s 65%.

http://www.pewinternet.org/Reports/2012/E-readers-and-tablets.aspx?src=prc-headlinehttp://www.emarketer.com/Article.aspx?R=1009106http://pewinternet.org/Commentary/2012/February/Pew-Internet-Mobile.aspxhttp://pewinternet.org/Reports/2014/E-Reading-Update/Tablet-and-Ereader-Ownership/Half-of-American-adults-now-own-a-tablet-or-ereader.aspx

JANUARY 2012








JANUARY 2014






September 2012http://www.asymco.com/wp-content/uploads/2012/08/Screen-Shot-2012-08-20-at-8-20-12.45.17-PM.png

More iPads in two years than Macs in 25 years.

Already talked about how many phones we’re selling.In Q3 2011, one out of every three Apple iOS devices sold during the last quarter was an iPad.14 million iPads versus 26.9 million iPhones.

As you can see from the curve, it’s actually getting steeper.Accelerating growth.

Who Owns What?

Smartphone eReader Tablet

18%

20%

19%

26%

16%2%

30%

21%

14%

18%

10%7%

19%

18%

17%

22%

13%

11%13-17

18-24

35-44

24-34

45-44

55+

Flattened out since a year ago, seeing big adoption at the older end. Was the more traditional 24-34 male.

http://blog.nielsen.com/nielsenwire/online_mobile/changing-demographics-of-tablet-and-ereader-owners-in-the-us/

August 2011

Age distribution

13-17 18-24 25-34 35-54 55+

17%

24%26%

14%

19%

7%

21%

33%

21%

18%

Smartphone Tablet

The last chart showed what percent of each age group owned these devices.

This one looks at actual users and shows the distribution of age.That is what percent of smartphone and tablet users are a certain age.

As you’d expect from last graph, see flatter distribution for tablets.

But here’s the point I want to make...

http://blog.flurry.com/bid/90987/The-Truth-About-Cats-and-Dogs-Smartphone-vs-Tablet-Usage-Differences

September 2011

Age distribution

13-17 18-24 25-34 35-54 55+

17%

24%26%

14%

19%

7%

21%

33%

21%

18%

Smartphone Tablet

Nearly 3/4 of smartphone users are 34 years of age or younger.[pause]


September 2011

Age distribution

13-17 18-24 25-34 35-54 55+

17%

24%26%

14%

19%

7%

21%

33%

21%

18%

Smartphone Tablet

But 2/3 of tablet users are 25 or older.These are grown-up, mainstream users. Not the typical early adopter.


September 2011

49%of tablet owners are women

Compared to 50% for smartphones and 58% for ereaders.

Evening out. This number was in mid 30s a year earlier.

My point here is that tablets are BIG. Not toys, not curiosities.

They don’t yet have the install base of smartphones,but growing MUCH faster.And in a more mainstream demographic.

http://www.pewinternet.org/Infographics/2012/A-Snapshot-of-Ereader-and-Tablet-Owners.aspx

You’re not “just” a visual designer.

That means you’re not just a visual designer anymore.

You’re anindustrial designer.

You’re also an industrial designer.

Or at least your job now includes aspects of industrial design,the art of crafting a physical thing.

Because when you design a touchscreen interface,you have real, honest ergonomics to consider.

In a very real sense, you’re designing a physical device.

Not literally.Virtual, flickering liquid crystals

But because they’re explored by human hands—unlike desktop experiences to date—you’re designing how hands interact

Been dealing with this on mobile for 5 years.

Phones and tablets confront us with a blank slate.Invite us to impose our own interface, ANY INTERFACE.

Because that interface requires touch,defines device in very physical way.

Soldering circuit boards, molding plastic, diecastReal ergonomics:Again how does it feel in your hand?

But to know that, you have to know how someonewill HOLD it in the hand.

Basically three ways to hold a phone. [next]

Steven Hoober did a field study,observed over 1300 people in the streettapping away at their phones.

Big plurality, nearly half, were tapping with one hand,with their thumb.

We switch between these grips often, very contextual.The most popular though is that one-handed grip.

Gives us freedom to use the other hand.To write, to hold coffee, to juggle a baby.

But I want to pause to look at that middle one, too,where we hold the phone in one hand and jab with the other.

[twitter]How do we hold our phones? Study by @shoobe01: 75% of time we tap w/ one thumb. 49% in one hand. http://j.mp/10nbnAD[/twitter]

49% 36% 15%

http://www.uxmatters.com/mt/archives/2013/02/how-do-users-really-hold-mobile-devices.php

Been dealing with this on mobile for 5 years.

Phones and tablets confront us with a blank slate.Invite us to impose our own interface, ANY INTERFACE.

Because that interface requires touch,defines device in very physical way.

Soldering circuit boards, molding plastic, diecastReal ergonomics:Again how does it feel in your hand?

But to know that, you have to know how someonewill HOLD it in the hand.

Basically three ways to hold a phone. [next]

Steven Hoober did a field study,observed over 1300 people in the streettapping away at their phones.

Big plurality, nearly half, were tapping with one hand,with their thumb.

We switch between these grips often, very contextual.The most popular though is that one-handed grip.

Gives us freedom to use the other hand.To write, to hold coffee, to juggle a baby.

But I want to pause to look at that middle one, too,where we hold the phone in one hand and jab with the other.

[twitter]How do we hold our phones? Study by @shoobe01: 75% of time we tap w/ one thumb. 49% in one hand. http://j.mp/10nbnAD[/twitter]

This picture shows a finger doing the tapping.Turns out that most of the time, when we usethis hold and stab posture, it’s not actually thefinger that does the work.

Instead, Steven observed:72% of the time people use the cradle grip,WE USE OUR THUMB.

75%of phone interactions

Add it up, and that means75% of our phone interactions are these two grips.

Three quarters of the time,WE TAP THE SCREEN WITH ONE THUMB.

We often talk about how we’re designing for fingers...

All thumbs.In reality, we’re designing for thumbs.As we’ll see, that truth cuts across other device types, too.

Thumbs are awesome.But they have their limits.

And this is it: the thumb-tapping hot zone.

Comfort zone for thumb when held in right hand.

Bottom of the screen opposite side of the thumb—left side for the right thumb—where tapping is easiest, where thumb naturally rests

Obviously thumb can reach anywhere onscreen,but this is most comfortable zone.

Important implications:That’s why iOS puts primary controls at bottom.

Turns desktop convention on its head

LEFT VS RIGHTSteven’s study found that two-thirds of the time,we tap the screen with the right hand.

But very fluid about which hand we use.More important bit is top vs bottom.

And look what’s NOT in the thumb zone: edit button.

Common pattern, though, is also to put a primary action up there.Post a new message always there in the top right of Twitter app.

You see a similar approach in the Foursquare app.Foursquare puts the checkin button in top right corner.Still the same idea -- put this data-changing button out of harm’s way.

More recently, moved the checkin bottom to middle.Hide navigation options.Single primary action in single primary area.

Instagram puts primary action -- add photo -- in the middle and highlights it.

Been talking about this in terms of thumb comfort,where your hand naturally falls on the screen.

But not only a thumb comfort issue.Fingers obscure view.This brings up an age-old principle of industrial design.Content at top, controls at bottom.

Here are a few examples.Feet at bottom, scale at top.

Dial at bottom, display at top.Controls below, content above.

Design to keep fingers out of way of content.Hands, feet, fingers at bottommake room for display at top.

Meat on the bottom, content on top.

Isn’t that right little professor?

Design to keep fingers out of way of content.Hands, feet, fingers at bottomto make room for display at top.

Consider this a cardinal rule:put controls below content.

That’s what you see in iOS.But it all depends on who gets there first.

So the story changes a little bit for Android.

Complicated by the presence of system buttons on the bottom.

IThey’re doing the right thing:controls below content.

The trouble is, where do you put YOUR app controls.

The operating system beat you to the bottom of the screen.Do you stack them on top like you see here?

Always bad to stack controls in touch devices—invites errors and mistaps—but it’s worst to do it at screen bottom.

Here’s the older version of Android—Gingerbread—that runs on mostphones right now. Hardware buttons at bottom.

There, the general guidelines was and remains:PUT APP NAVIGATION AT TOP,so it doesn’t interfere with those system buttons.

Trouble is that this necessarily means that you’re covering the contentwhen you go to tap that button.

Take a look at the same legacy version of the Foursquare app in iPhone...

Navigation bar back on the bottom.And at top right: “check in” which affects data,safely tucked away at top right.Available but out of danger of mistap.

Got to move it to the top to get out of the way of the menu button at bottom. Android has a *lot* happening at the bottom of the screen.

Ergonomically and visually better to have main controls at bottom, but Android already has lots there.

This is changing in Android 4, Ice Cream Sandwich...This style of tab is going away, and so is the option menu.This is what we get instead...

The recommended default navigation for newer versions of Android,Ice Cream Sandwich and Jelly Bean,uses the ACTION BAR pattern shown here.

Again, at screen top.All to avoid that stacked navigation at screen bottom.

So that’s Android.

And the web?

Turns out the story changes yet again for touch on the webfor phones and small screens.

3 problems pinning a navigation toolbar to either top or bottom.

1st is that you would normally use position:fixed to do that.Really pretty buggy. It doesn’t work at all on some browsers,other browsers it’s implemented poorly.

So you have a technical challenge of just getting your navigation barto sit still on top or bottom of the page.

Second, you start to choke out the content.Barney’s mobile website.

Logo and toolbar are fixed on screen.Looks okay in portrait, even though over 1/3 of screen isbrowser chrome, logo, and toolbar.

[next: scroll] so you’re giving up a lot of real estatethat you could give to content.

But much worse when you flip to landscape,barely a sliver of content peeks through.

It’s barely a Blahnik!

And worse, try to scroll by swiping on that fixed toolbar,and you can’t. doesn’t respond.all the interaction AND content is limited to that tiny strip.

----

The screen already has browser chrome.Fixing your own navigation tools on screen makes it worse.

Nobody comes to your website to use yourawesome navigation controls. Get them out of the way,FEATURE THE CONTENT.

3rd problem is also related to browser chrome.In iOS, Safari has a fixed toolbar at screen bottom.

Not all browsers do—Android doesn’t—but in this case,putting a toolbar on the bottom here would create that stackedtoolbar problem.

So three strikes against putting controls anywhere permanent on the page.

Instead, this is my favorite pattern for small screens.

PHONE GUIDELINESiOS appsControls at screen bottomAndroid appsControls at screen topWebControls at page bottom

Started with very simple concept:Content at top, controls at bottom.

But it turns out, environment matters.Who gets there first.

And as a result, you get three different rules for these different platforms.

millimetersSEVEN

BUT HOW BIG, Josh?When we’re designing for touch, how big should touch targets be?

7mm, about ¼”Spread of fingertip as contact point on screen.Size of target that finger can reliably hit

missed taps25%

20%

15%

10%

5%

target size 3mm 5mm 7mm 9mm 11mm

1 in 30 (3%)missed taps

1 in 100 (1%)

1 in 200 (0.5%)

http://msdn.microsoft.com/en-us/library/windows/apps/hh465415.aspx

Microsoft did some research for Windows—for both desktop and phones—and found very consistent results.

So 7mm pretty good for everyday, 9mm if you’re being super-cautious.

I don’t know about you.For me, mm isn’t exactly my standard css unit.So how do we specify this size?

Well maybe standards can help us out here.

[twitter]How big should touch targets be? Microsoft study: 7mm good, 9mm to be safe. http://j.mp/UzDGfZ[/twitter]


css 2.11 pixel = 1/96 inch7mm = 26.457 pixels

30pixels

Starting with CSS 2.1,pixel measurements don’t refer to dots on the screen but an actual physical distance.

1 pixel = 1/96 inch, like the old 96dpi monitors.Let’s round it up, call it 30 pixels. Done, right?

Unfortunately, no.

It SHOULD be 30 pixels, if all was neat and clean in the world.

This doesn’t factor in the practical reality of dynamic viewports.You know these from your phones and tablets.This is the magic that lets you see a full-width website on a tiny screen.Pinching and zooming.




Unfortunately, no.




DynamicViewports



Unfortunately, no.



Launch the NYTimes and you get this.Where’s my 96 dpi pixel now?This is 972 pixels, but it sure doesn’t look like like 10” here.Squeezed into less than 3”.And that’s a good thing—the only way I can see the whole site.From there, I can zoom in to read it and it’s suddenly, much larger.

Or this [next] or this or this.

There’s seemingly no “true” size for this thing.

By design, dynamic viewports destroy the very ideaof mapping a pixel to a fixed physical size.

But all is not lost.Devices do still report their device width.That’s basically their native, “true” size.

These are the numbers our media queries hook onto.

[twitter]CSS standard says pixel is 1/96 inch (96 dpi), but dynamic viewports mean it’s rarely the case. Vexing viewports: http://j.mp/WSj7vt[/twitter]

320pixels

iPhones for example, report their device width as 320 pixels.

That means that, unless you specifically say the page should be wider,your designs will default to a 320 pixels wide on an iPhone.

This has nothing to do with how many physical pixels are actually on the screen.Unrelated to retina display.

These are called virtual pixels. Call them web pixels if you like.Here, iPhone says, it has a device width of 320 web pixels across in portrait.Even if, in practice, a retina iPhone has 640 hardware pixels across.

[next]

On iOS, that fixes a web pixel at around 160 dpi,not 96 as CSS spec suggests.

Apple popularized this approach to the dynamic viewport,and most platforms and devices have followed suit with similar dpi numbers.

So. Going with Apple’s recommended touch target turns out to do the trickin most cases.

Forget about the “proper” 30 pixels we calculated earlier.

320pixels

On iPhone, again, that will be 320 web pixels.

So it turns out that, on individual devices at least,web pixels do have a fixed width,assuming you set the viewport to the device width.

It’s just that it’s almost never the 96dpi sizethat the web specification says it should be.

Apple’s decision to set the iPhone device width to 320px...

[next]

...made iPhone’s web pixels 160dpi, and that choice impacted all the mobile browsers that followed.





160dpi

On iPhone, again, that will be 320 web pixels.

So it turns out that, on individual devices at least,web pixels do have a fixed width,assuming you set the viewport to the device width.

It’s just that it’s almost never the 96dpi sizethat the web specification says it should be.

Apple’s decision to set the iPhone device width to 320px...

[next]

...made iPhone’s web pixels 160dpi, and that choice impacted all the mobile browsers that followed.





44pixels

Boom.Forty-four pixels, friends: that’s the new hotness.

7mm in most touch browsers.

Now. Most of us don’t work in pixels any more.

2.75ems

Boom.

Based on the 16-pixel default size,2.75 ems should give you a good touch target.

[twitter]For the web, 44px is a sturdy minimum size for touch targets, or 2.75 ems.[/twitter]

44

Compromise necessary sometimes.Have to get all the letters of the keyboard on the screen.Squeeze to 29 width.

In iOS, 44 number is not just a guideline,but it’s actually cooked everywhere into the operating system.

4429x

As long as one dimension is at least 44,can squeeze other to 29.

Practical minimum for tap targets: 44x29 or 29x44.

44

44

44

Again, though, 44 appears everywhere.Nav bar, height of rows in standard list view, tool bar.

88

88

Home screen grid doubles up to 88.

Idea is that organizing in multiples of 44multiples of finger’s touch surface,app not only looks rightbut literally feels right

Design to a 44 pixels rhythm.

Gives you easy, to read, easy to touch interface.

How cool is this!All elements sized exactly in proportion to your finger print.

For the hand, but also

of the handNot only for the hand, but of the hand

Every element in proportion,Not only to one another but to the finger itself.

Design to a 7mm rhythmDon’t think 7mm just for buttons but for overall layout

That’s lovely.But THIS is not....

Man enough: had one of these bad boysCasio Gold Data Bank watch. Calculator watch.Not only small buttons but too closeAim for 5, get 8 or 2More wheel of fortune than calc

Not just button size but spacingCloser -> larger

7mmtouch targets

2 mmpadding

For example, this is a terrible idea.It is both awesome AND terrible.

The Triple Triple burger from a place called Jake’s Wayback Burgers.Nine patties on one sandwich.Over 5000 calories.

Too much of a good thing is still too much.

Removing stuff from an overloaded webpage can be good.It has a clarifying effect.

Because in general, and certainly for mobile interfaces...

[twitter]Too much of a good thing: Triple Triple Burger packs 9 patties (and 5100 calories) into one sandwich: http://j.mp/Z2tjht[/twitter]

This is the Accuweather.com iPad app,but I’m not sure that it manages complexity in the right way.First of all, scary.

But more important, jammed with information.More than you need at any given time.Don’t make me scan all this data for what I’m looking for.Instead, let me ask for it as I need it.

Manage complexity not by presenting it all at once,but by managing it through give and take.

Accuweather.com.Actually does a better job with complex informationin small screen of iPhone app instead of iPad app.

Dense info for the current moment

[next]Oh baby... yeah... Now we’re talking. Ken’s in hog heaven.I’ve got dew point, I’ve got humidity, barometric pressure.

Nice start:But how to provide all of the day’s detailed info?

Swipe at current conditions to move into future. [next]

Detailed conditions for 7am, and temps for later.I ask app about 10am… by touching 10am!Only when I ask for that info does it give it to me.Question, answer. Ask, receive.

This is true in all interfaces, but especially in mobile...

Claritytrumps density

In a mobile interface, clarity trumps density.

Requires more taps than just dumping all the data on you directly.But each screen more digestible.

Progressive disclosure

Progressive disclosure. A little bit at a time, as people need it or ask for it.

Progressive disclosure helps you uncomplicate complexity.

You can trim some things off the screen and return to them later.

NOT EVILExtra taps & clicks

The web has given us a squeamishness about extra clicks.Network latency.

[slow]In mobile: tap quality far important than tap quantity.

As long as each tap delivers satisfaction,extra taps are ok. ARE GOOD.

It invites conversation,give and take that you can get at and explore.

Speaking of not being evil...

Speed matters.

We think about this in terms of performance more often than we think about it in human efficiency, but some of this stuff just doesn’t work very quickly on touch.

40%jump shipafter 3 seconds

Source: http://bit.ly/o59qCISource: http://e-commercefacts.com/research/2011/07/what-usrs-want-from-mobil/44

33% morejump when go from2 to 8 sec response


80%would use mobilemore if it was faster

This isn’t just about how fast your app or website loads.This is also about how long it takes to slog through your interface.

USER PERFORMANCE also matters.How long does it take me to complete the task?How many swipes and taps?

Let’s look at one industry example.


taps52seconds102

taps40seconds109

Hotel Tonight

My friend Luke Wroblewski turned me onto this company,which takes a wholly different approach.Total focus on speed

taps/swipes4seconds8

The focus on a simpler task:I want a hotel tonight for this town. And they ruthlessly trimmed the amount of interactions it takes to to choose and book.They use sensors. They use account history.Boom boom boom. Three taps and a swipe.

If you want to sell stuff on eBay, there are a LOT of forms to fill out.You’ve done a great job of shepherding people through those,making them efficient, some of the forms are optional.but there are still more opportunities to trim those down,speed your way through, and we’ll explore that this afternoon with mobile team.

I want to be careful here, though.Because more taps aren’t necessarily bad. And fewer taps aren’t necessarily good.

A risk of being ruthless about this is that you can trim useful content and features.

So the goal here is not necessarily to reduce tapsbut to reduce GARBAGE taps.Interactions that don’t add anything.

Otherwise, the risk is that you’ll cut muscle, not fat.

Speed matters.

So as I said, speed matters. It’s a competitive advantage.

And if speed is so important...

Speed kills.

...that means that it should also bump aside other considerations.

Speed should eliminate certain interface conventions in touchscreens.Let’s see what speed kills when you design for touch.

Kill select menus

*Speed kills the select menuLots of taps. Entering states: takes longer for me to choose New York in a list of 50 itemsthan it does to type NY. Let me do that.Or give me a shorter single-tap menu, guess-ahead fields, steppers, suggestions.

* Kill extra fieldsIf you have a zip code, you don’t need to ask city and stateSquare field example

Kill select menusKill extra fields

*Speed kills the select menuLots of taps. Entering states: takes longer for me to choose New York in a list of 50 itemsthan it does to type NY. Let me do that.Or give me a shorter single-tap menu, guess-ahead fields, steppers, suggestions.

* Kill extra fieldsIf you have a zip code, you don’t need to ask city and stateSquare field example

For example, credit card info, what’s the minimum you need:

Credit card numberSecurity codeExpirationBilling zip

Square pioneered shmooshing all of this into a single field,using just the single numeric keyboard.Very fast. Just tap through the numbers,in a single field with minimum info

Zachary Forrest built this for the web,a Square-inspired widget for accepting credit card forms.

http://zdfs.com/toscani/paymentInfo/index.html

Video from @lukew: http://www.youtube.com/watch?v=XUa5pfHdVwQ

Kill select menusKill extra fieldsKill confirmation dialogs

* Kill confirmation dialogs

Confirmation dialogs don’t work.Too easy to ignore, too annoying to be seamless.We motor through them without paying attention,undoing their purpose.

But they’re still a speedbump, an interruption.How can we make that confirmation faster and more organic?

jiujitsuGESTURE

One last topic on awkward gestures, something I call gesture jiujitsu.Art of self defense against mistaps.

The problem:Ease/sensitivity of touchscreen can work against you.Buttons get pushed when don’t intend it.Surprise calls from handbags and backpockets.Carelessly delete data by tapping wrong item.

Awkward or challenging gestures can protect against mistaps.

Swipe requires intention.

Cross something out to delete or mark it read.Swipe to unluck.Swipe to answer the phone.

Or combination of taps.In any case, a brief moment of focus, that’s not a confirmation dialog.

Kill select menusKill extra fieldsKill confirmation dialogs

* KILL THE KEYBOARDHow can you get away with preventing the keyboard from popping up at all?

Select menu, mentioned it’s faster just to type than to select.But that’s still not ideal, that’s still a four-tap operation, just to type two letters.

To enter an airport code: show airport options from history or preferenceTo enter a date range, just drag or pinch the range, no need to fuss with buttons.

Kill select menusKill extra fieldsKill confirmation dialogsKill the keyboard




Kill select menusKill extra fieldsKill confirmation dialogsKill the keyboardKill detail screens




Again progressive disclosure.

Show enough summary information that you may not need to drill down.

Kill select menusKill extra fieldsKill confirmation dialogsKill the keyboardKill detail screens

* KILL CAROUSELSThis might seem counter-intuitive because carousels are a kind of progressive disclosure.

You see this in editorial and marketing sites. Want to have high-profile visuals for lots of stuff, but how do you cram it in?

But: faint scent of information. What’s on the next slide? What’s to pull me through?Worse, it takes nine interactions to see that tenth slide.

Kill select menusKill extra fieldsKill confirmation dialogsKill the keyboardKill detail screensKill carousels

* KILL CAROUSELSThis might seem counter-intuitive because carousels are a kind of progressive disclosure.

You see this in editorial and marketing sites. Want to have high-profile visuals for lots of stuff, but how do you cram it in?

But: faint scent of information. What’s on the next slide? What’s to pull me through?Worse, it takes nine interactions to see that tenth slide.

I led the design of People magazine’s responsive mobile website last year,worked with Ethan and Karen on it.

Take a look at this latest news area. Lots of headlines.[next]As it compresses we condense those headlines into a carousel.

Balances content density with quick ergonomic and cognitive scanning.

Carousels are great for showing panels of several content elements; a read-more short cut inside the page.

They’re bad for featuring one item at a time, where you hide the content and make it so that it’s not findable.

But panels of a few items at a time are great for this purpose.

Stack sections....And then we have these carousels, which are visually hidden and fly in from the side.

This of course takes JavaScript. So what we’ve done here is enhanced the content experience with a browser-specific feature. JavaScript support.

3 6 9

Stack sections....And then we have these carousels, which are visually hidden and fly in from the side.

This of course takes JavaScript. So what we’ve done here is enhanced the content experience with a browser-specific feature. JavaScript support.

3

MORE

Doesn’t have to be carousels. “More” buttons to expand into view.

3

6

9

Doesn’t have to be carousels. “More” buttons to expand into view.

NOT EVILExtra taps & clicks

The web has given us a squeamishness about extra clicks.Network latency.

[slow]In mobile: tap quality far important than tap quantity.

As long as each tap delivers satisfaction,extra taps are ok. ARE GOOD.

It invites conversation,give and take that you can get at and explore.

Speaking of not being evil...

EVILLet’s talk

Our mission is to save the kitten.Save the kitten!

How can we solve this with progressive disclosure.

The LongScroll

Our mission is to save the kitten.Save the kitten!

How can we solve this with progressive disclosure.

Luke calls these off-canvas patterns.Idea is pretty straightforward:

Instead of stacking your columns into one long column,push some content offscreen until it’s needed.

Secondary content is pushed a screen away,summoned with a tap or a swipe.

You might do it like this...

[twitter]“Progressive disclosure” manages complexity by revealing it gradually. @lukew’s off-canvas patterns helpful: http://j.mp/XoaUxI[/twitter]

or like this...

Kill select menusKill extra fieldsKill confirmation dialogsKill the keyboardKill detail screensKill carousels

* KILL TOUCH ITSELF

This might be a strange thing to say in a touch workshop.

But really that’s what this whole section is about: how can we reduce the number of interactions required to satisfy user needs?

Kill select menusKill extra fieldsKill confirmation dialogsKill the keyboardKill detail screensKill carouselsKill touch itself?

* KILL TOUCH ITSELF

This might be a strange thing to say in a touch workshop.

But really that’s what this whole section is about: how can we reduce the number of interactions required to satisfy user needs?

theBEST TOUCH

interface

is sometimesNO TOUCH

at all

I’ve realized that the best touch interfaceis sometimes no touch at all.

How can we save the user from tapping into the screen at all?That’s the point of barcodes, of computer vision,of speech recognition.

Touch is just one of an emerging set of interactions.

Touch matured first, but speech and natural gesture are set to pop, too.Facial recognition. All the ways we communicate as humans, starting to come through

All of this stuff is still a little unreliable.Siri is in beta.Kinect great for games, but wouldn’t want to run a nuclear power plant with it.

These are the things that we’re going to have to start designinginterfaces for. For speech, for gesture, for facial recognition.

As we think about designing for tablets and phones,we have the opportunity to push interaction off the screenand into the environment around us.

Sensors let us do this. The superpowers on board these devices.Think about camera vision, about shazam-like audio processing.About GPS. Where the devices can make smart inferencesand do the input on our behalf.

That’s the topic of my keynote tomorrow afternoon.So I’ll spare you that content for now.

Kill select menusKill extra fieldsKill confirmation dialogsKill the keyboardKill detail screensKill carouselsKill touch itself?

So those are some speed measures you can take.

How you doing?

We’ve been talking about phones a lot for touch.

And that’s just because that’s where the mainstream knowledge and experience with touch comes from, for both designers and consumers.

But this is after all a tablet conference... so...

74%http://www.elearningguild.com/research/archives/index.cfm?id=174&action=viewonly

Hoober and Patti Shank found that 74 percent of phablet owners work their screens with two hands.

Compared with just 51% on smaller phones.

35%In another study, Hoober and Patti Shank found that 74 percent of phablet owners work their screens with two hands. The most-used grip (35 percent) has people holding their phablets in one hand while tapping with the index finger of the other.

59%thumbs on phablets

Choke up to get at higher areas on the phone.

No matter what your handhold, the top of a phablet screen is always going to be a difficult reach, particularly for one-handed use.

As with smaller phones, it’s ideal to follow the “content above controls” rule to keep frequent tap targets within reach and out of the way of content.

The change here is for Android;

instead of putting all Android controls at screen top as you would for smaller phones, phablet interfaces can slide some of those frequent controls down to screen bottom by following the split action bar design pattern.

The split action bar moves some frequent controls out of Android’s standard action bar down to a separate toolbar at screen bottom.

A floating trigger button can be a useful workaround here. These buttons tuck into the bottom corner of the screen, and stay fixed in place, hovering above the the rest of the interface as it scrolls past. Because this button doesn’t extend full width, the stacking penalty on Android is far smaller than it is for a full toolbar. In Android’s UI lingo, this trigger button is called a floating action button; the platform’s design guidelines for the button suggest that you space it far enough from the system buttons that the stacking risk is minimal.

A floating trigger button can be a useful workaround here.

These buttons tuck into the bottom corner of the screen, and stay fixed in place, hovering above the the rest of the interface as it scrolls past. Because this button doesn’t extend full width, the stacking penalty on Android is far smaller than it is for a full toolbar.

In Android’s UI lingo, this trigger button is called a floating action button; the platform’s design guidelines for the button suggest that you space it far enough from the system buttons that the stacking risk is minimal.

When my team designed Entertainment Weekly’s responsive mobile website, for example, we used a floating trigger button to offer quick access to sharing tools for all screen sizes from tiny phone to large desktop. Tap the button once to reveal a toolbar of sharing options.

Controls/swipes at bottomAnchor-link web navAvoid cross-phablet stretchBring the mountain to Muhammad

Phablets

In addition to tabs at top, swipe through views.

Anchor link nav: The top-corner position staked out by a menu link will admittedly be out of reach for a single thumb, but it also turns out that navigation menus are only rarely used by web users.

Cross-phablet stretch:Avoid tap targets that hug the left or right edge too tightly. People love that middle of screen. In the main body of the page, favor targets that stretch at least a third into the screen, if not the full width.

Mountain to muhammad:Most interfaces are fixed landscapes that we roam across. We move to the buttons; they never come to us.Samsung, for example, created a special “one-handed operation” mode for its jumbo Android phonesApple’s reachability.

Instead of sliding the entire interface up or down, though, a more practical use in webpages is to slide menus in from the edges.

As mentioned earlier, though, side tabs are less than ideal for phablets, where side edges are both outside the comfortable thumb zone. Sides are still far easier to hit than the top, making this sliding panel solution better than top-of-screen controls. (Better for tablets)

?What about iPad and other tablets?

Unlike phone, iPad doesn’t have a conventional grip.Phone has three grips.

iPad: Hold it in all different ways.Much of that depends on stance.Standing, desk, couch, bed.

In general: two handed, side, medium to bottom grip.You’ll hold it higher if standing, lower if resting it on your lap, and so on.

But across the bottom is bad for controls.* Rarely going to hold it at the bottom. Floppy.* Out of reach of thumbs* Out of your sight line. Unlike phones, not at a glance.* And for me, a more pressing problem:

Barcalounger with bag of cheese puffs.

Screen bottom thumb zone for phone doesn’t apply.But thumbs matter. You still want to optimize for thumb use.They just fall in different place.

Back to basics: avoid encouraging covering content.Generally want to avoid putting controls above content.

Here’s The Daily, News Corp’s iPad-only daily newspaper.There’s this scrubber control at the top...

So you tap the scrubber, nice visual table of contents.

And you can drag scrubber left and right to move through the pages.

Trouble is, hand blocks the thumbnails you’re browsing.

Danger of putting controls at the top center of the screen.Never put controls immediately above the display.

Can swipe thumbnails instead of using the scrubber.So, problem solved?Because launch the display by touching the scrubber,not obvious that you can navigate pages directly.

Best practice: avoid controls at top middle of the screen.

Instapaper.Here you see controls at the top of the screen.Closer to desktop standards we’re used to.More visually available at the top.

Turns out it’s better ergonomically, too.

[next]In general, your grip will tend to be around middle of screen,so thumbs at top 1/3.

You want to avoid putting controls atcenter where your hand roams over the content, obscures it.

Instead, put those controls in reach of where thumbs come to rest.Don’t make me haul my giant meat pointer.

Time magazine’s Populist app one of few in publishing industrythat places buttons in ergonomically comfortable position.

Howcast. Content optimized for landscape viewing.For landscape, consider controls at side,where you’ll be holding it, and you have plenty of room.

Our Choice app.Some bottom of page stuff is fine,especially when you’re working on somethingthat will change the content in the main frame.

Not ideal for quick access to tools, or frequent taps.

But when you need a controlto do browsing or action on a screen, bottom works well.

iPad and other large tabletsFavor sides and corners

Avoid top center

Use bottom to seeeffect in canvas

All good?We’re just getting started. Because friends...

“The Hybrids”Now we’re onto something at once thrilling and awful. The greatest horror our screens have ever experienced.

Oh yes, friends, it is a terrifying combination indeed.

Oh yes, an unholy alliance.

Touchscreen AND keyboard.Sometimes touchscreen detachable to make it a tablet.Sometimes fold it back to make it keyboard free.Other times it’s always attached.

All combine keyboard and touch at some point.

40 models of touchscreen laptops and ultrabooks on the market now.And more to come.

Intel has announced it’s require all ultrabooks using its latest chipsto include touchscreens.

How long before that happens to laptops, too?

We’ll see if this proves popular but a TON of these are becoming available,more on the way.

Similar devices have been around for a while, but not in large numbers,so research and design practices are a bit rare.That said, some ergonomic behaviors are already clear.

1. Mouse/trackpad use drops way off.2. You get people touching directly almost all the time.3. Scrolling, selection, even form fields.

People accustomed to tabbing between forms switch,touch fields directly even if less efficient.

Turns out that touch is a powerful invitation. people use it,

But wow, that seems physically uncomfortable, right?Steve Jobs always said nobody will ever want to use one of these screens.Nobody wants to work like this, with gorilla arms.

And it turns out people actually DON’T do gorilla arms.

[twitter]Intel says: Ultrabooks must have touchscreens to use latest Intel chips. How long before laptops? http://j.mp/YC1D36[/twitter]

One or both hands gripping the bottom of the laptop.If one hand, using the other to tap or scroll with index finger.

Support for floppy cover.So this is the hot zone.

This is why in Windows 8 you have these gesturesto swipe toolbars in from the right side or from the bottom.

Extremely comfortable for thumbs in this position.

Confusing things a bit is the fact that using index fingerto work the heart of the canvas means this is the thumb zone.Corners are tough.

EXACT OPPOSITE OF THE THUMBZONE.

So what to do? navigation and controls at bottom and sides,or at top and middle?

It turns out after frequent use, start to go all thumbs.Use thumbs for all taps, stretching in.

The thumb dance.Do that crazy dance, little guys.

So this thumb pattern tends to be ideal for frequent controls.

How does this compare with a tablet’s hot zone?If you overlay the two...

You get a common area that looks like this.

So the most touch-friendly zones for layout—addressing both tablet and dektop—are the sides.

I don’t know about you...desktop designs I’ve built haven’t optimized for these areas.

We generally optimize for top or middle of screen, right?But top middle is actually most hostileto this thumb zone.

Getting the picture?For all of these form factors, not about making finger-friendlyBut THUMB FRIENDLY.

All right.---Like I said earlier: we typically focus on the visual.It’s not yet in our industry’s DNA to considerphysicality and environment in our layouts

That’s why:many look at their legacy desktop layout on iPad:yep, that works. Roughly the same size.IT LOOKS OKAY.But that rarely means it’s also touch-friendly.

Tiny menus,hover events,eensy weensy tap targets,bad ergonomic position.

A headache forresponsive design

This actually reveals a flawin the way most of us have handled responsive design.

Its whole purpose is to present different layouts for different screen sizes.It’s a visual device, a layout device.

But we’ve also been using it as a way to detect touch.Small screen: touchBig screen: cursor

Screen size is a

lousy wayto detect a touchscreen.

YOU GUYS:Screen size is a lousy way to detect the presence of a touchscreen.

It was a bad assumption to start with,but it’s getting worse as touch spreads to bigger devices.

So what, then?Turns out: there’s no reliable client-side wayto detectif it’s a touchscreen,a keyboard-and-mouse,or a hybrid.

Could use javascript to detect a touch event object,but not all touchscreen browsers have touch events.Strange but true.

So for now, we’re just guessing.Might be a touchscreen. Might not be.

[twitter]Faulty thinking: it’s a big screen, so it must be mouse/cursor, not touch. But that’s how we operate now.[/twitter]

CSS4 media queries@media (pointer:coarse) {

}

input[type="text"] { min-height: 44px; }

But even that doesn’t get us there.

What about devices like Microsoft Surface, which have BOTH touch and a touchpad?What does that even report?

So even here we’ll have uncertainty.

For now, though, since we can’t know whether a device is touch or not,we have to assume it is.

We have to support the lower-resolution pointer, the finger.

As far as I’m concerned,that leads to a pretty sturdy principle going forward.

[twitter]CSS4 will introduce a “pointer” media query to detect touch vs cursor UI. http://j.mp/151PWIr[/twitter]

All desktop designs have to betouch-friendly.

That’s what Windows has done for Windows 8.Have to.Huge challenge for Windows: designing for every possible input.

Need to rethink some “settled” design patterns.

[twitter]All desktop designs have to be touch-friendly now: http://j.mp/Tt4h87[/twitter]

So we’re going to have to do a little mind transfer.

Shift some perspective that we’ve developed for phones and tabletsover to desktop interactions, too.

hover eventsRest in peace,

Rest in peace, hover.We’re retiring you, at least for crucial tasks.When you design for touch, you have to treat hover as an enhancement.It can’t be the only way to get at content.

Thumb controlsat left side

Left side because that’s where right-handed users will support the device.

We’re beginning to see some patterns that are kind to fingers.

ZDnet offers these share controls and always keeps them in view when you scroll.

Very thumb friendly position right at the left edge.

This is the pattern I recommend:put frequent controls at left of the screen

For right-handed users:grip with the left hand, jab with the right.Means your left thumb always there.

NOT main navigation.Put something there that people actually use.Usage patterns bear out the fact that:NAVIGATION MENUS ARE A MISERABLE LAST RESORT

We slouch over to main navigation only when the rest of the page has failed us.when we can’t find what we want in the main canvas.

So it’s fine to keep main navigation in that hostile top-of-screen.Instead, concentrate on the controls that people actually, frequently use.Actions people want to take on the page.Share buttons a good pattern here for content sites.

missed taps25%

20%

15%

10%

5%

target size 3mm 5mm 7mm 9mm 11mm

1 in 100 (1%)


Microsoft did some research for Windows—for both desktop and phones—and found very consistent results.

So 7mm pretty good for everyday, 9mm if you’re being super-cautious.

I don’t know about you.For me, mm isn’t exactly my standard css unit.So how do we specify this size?

Well maybe standards can help us out here.

[twitter]How big should touch targets be? Microsoft study: 7mm good, 9mm to be safe. http://j.mp/UzDGfZ[/twitter]


44pixels

Boom.Forty-four pixels, friends: that’s the new hotness.

7mm in most touch browsers.

Now. Most of us don’t work in pixels any more.

2.75ems

Boom.

Based on the 16-pixel default size,2.75 ems should give you a good touch target.

[twitter]For the web, 44px is a sturdy minimum size for touch targets, or 2.75 ems.[/twitter]

New desktop design guidelines

Hover is an enhancementBottom left for controlsBig touch targets44px rhythmProgressive disclosure

designing forGESTURE

Super excited about possibilities of touch interfaces.

I believe forces—or should force—important,FUNDAMENTAL changes in howwe approach the designs of these interfaces.

Touch will help us sweep away decades ofbuttons—menus—folders—tabs and administrative debristo work directly with the content.

When you remove this thing—the cursor, and the mouse, too,these prosthetics we've pointed at stuff for 25 years—all that remains...[next] is you and the device, or better...[next] you and the content.

Or that’s the illusion we’re able to create.Changes the experience when it feels like you’reworking directly with content and objects.

It cuts through complexity to interact directly instead ofpumping buttons.

Buttons do add complexity. It’s a problem we always confront as designers.The more features you add,the more interface comes along with it.If we’re not careful, we start drowning in interface chrome.

It’s even in cars.

Content

When you remove this thing—the cursor, and the mouse, too,these prosthetics we've pointed at stuff for 25 years—all that remains...[next] is you and the device, or better...[next] you and the content.

Or that’s the illusion we’re able to create.Changes the experience when it feels like you’reworking directly with content and objects.

It cuts through complexity to interact directly instead ofpumping buttons.

Buttons do add complexity. It’s a problem we always confront as designers.The more features you add,the more interface comes along with it.If we’re not careful, we start drowning in interface chrome.

It’s even in cars.

http://bit.ly/civic-buttonshttp://www.youtube.com/watch?v=Y7cXNOZw5YE

We’re adding more complex—and frankly inept—interfaces to cars, often touch interfaces,AND IT’S DANGEROUS.

Mike Monteiro:Now that we’re putting bad interfaces behind 3000 pounds of metalwe’re going to see our first caseof a designer going to jail for bad UI.

There are real consequences to complex or cluttered interfaces.

But let’s start with a more modest and less serious example.One that shows how traditional controls and buttonsdon’t work very well on touch screens—especially as those touch screens become larger.

Tablets. They’re a big deal.By end of the year, 30% of US adults will have a tablet.One in three. Incredible growth.In two years, Apple sold more iPads than they sold Macs in 25 years.Windows 8 prompting device makers to flood marketwith touchscreen laptops and tablets.

We have to get our touchscreen tablet mojo on.

[twitter]Too many buttons in cars: http://bit.ly/civic-buttons (It’s funny, but: complicated UI in cars is dangerous stuff.)[/twitter]

Traditional controls feel confining on larger touchscreens like iPad.

Tiny button in the top left of iPad apps not easy enough to hit,yet I’m asked to hit it all the time.

Lots of iPad apps follow pattern you see in mail app.The split view for navigating layers of content.So I’m reading this riveting email from Facebook.

[one tap]

But I want to pop out to my sent mail.I have to hit this tiny target,navigate this series of buttons.

Big expansive screen, but I’m constantly required topeck at this tiny little patch in the corner.

I hatethe iPad’s back button

with the heatof a million suns

Even though iPad UI buttons same size as phone,takes more effort/concentration to hit them.

Enormous screenPeck at tiny block of pixels in top left.UI split view pattern = constant peckingMeat pointer.

Let people be lazy

This should be our mantra as designers.Let me be as lazy as I wanna be.

Coarse gestures,not fine-tuned pecking.

Do you even need buttons at all?Or can you at least provide an alternative?

iOS5. Instead of hitting that inbox button to see split view.

Swipe anywhere on the screen to pull out gesture.A supplement, an alternative to button.

And we can go beyond simple swipes and taps.Another approach might’ve been 5-finger touchto bring up that popover.

There are issues here:Discoverability, which I’ll talk about in a bit.

And also accessibility.I mean hey: Not everyone has five fingers.

So again, gestures are alternatives, supplements.

Gestures are thekeyboard shortcuts

of touch

Gestures are the keyboard shortcuts of touch.Often supplements to old-school buttons.

What makes them shortcuts?It’s the timesaving aspect is that you can just slap at the whole screen.Again: coarse gestures instead of fine-tuned tapping.

Reeder

Reeder, a feed reader for Google offers a good example.

Reeder for iPad provides a good example.News feed reader.

Fast, quick action that uses entire screen as the control.

Pinch and spread being used not just for physical zooming,but for navigational zooming. Moving up and down the hierarchy.

Still has back button. But...Do you even need that back button at all?

[twitter]Emerging pattern: semantic zoom. Pinch in and out of hierarchy. E.g. pinch to close detail view, return to list.[/twitter]

Storehouse

Reeder, a feed reader for Google offers a good example.

Twitter

Twitter for iPad managed to eliminate back button entirely.

The app designed to really encourage exploration.Interact directly with words, URLs, entire content contexts.

[flip through]

Tap the content, slide these views around.Paw thru your history.Each slide is a representation of the content.You’re sliding the content back and forth, like cards.No back button needed.

Key bit to designing for touch is imagine:how would data objects look and behaveif they were suddenly physical? slips of paper on my desk.

Big screens invitebig gestures

[slow] Big screens invite big gestures.Don’t have to hit the little button.Let me just swipe at the whole thing to take an action.

I’ve been talking about this in the context of ergonomics and concentration.That it’s harder to hit a button than paw at the screen.

But there’s also a more fundamental conceptual issue here.

HACKButtons are a

In the real world AND in software, buttons are abstractions.Work at a distance on the primary object.Often necessary, the best available solutionLight switches.Operate at a distance. Add a middle man, an extra layer of complication.

Touch: get rid of visual abstractionsto work with content directly.

Not saying they’re evil or bad.

Even Microsoft is getting this religion.Windows as you know it is being killed this fall.“Do more with less”“Authentically digital” (natively digital)

Replaced by an entirely new OS that only happens to include a Windows 7 emulator.

Windows 8 is a touch-based OS.Designed from bottom up for touch FIRST.

Apple’s been crowing about the post-PC era,but it seems to be Microsoft who’s doubling down on that bet.

http://bit.ly/win8-login[twitter]New inputs let you revisit sacred-cow interactions. See gesture-based password replacement in Windows 8: http://j.mp/win8-login[/twitter]

Whither the Web?

Before I go much further, I want to talk about wherethe web stands in this new wave of interface innovation.

This new class of devices creates interaction expectationsthat browsers frankly aren’t very good at.

Jen and Luke both showed us emerging APIs that willmake browsers much more capable. Lots of great accessto device.

But when it comes to touch gestures,have a long way to go.

Two big barriers here.

It’s hardto code gesturesin JavaScripttouchstarttouchmove touchend

First, real support for gestures.Because out of the box, JS coders have really weak broth to work with.Look at this bullshit.[next]

Are you kidding me?These are the events we get.Coders forced to start w/most primitive elements.

2-finger touch? Code it from scratch!Detect the touch, location, motion... design entire gesture.

And that’s for the browsers that even support it.Windows Phone: no touch eventsSymbian: sorry.

WebKit and Opera Mobile, that’s it.

We should have better, more abstracted gesture eventsto easily get at pinches, at multi-fingers touches and swipes.

So one job we have is to agitate for better gesture support.Challenging enough to design complex gestures on native SDK.Far worse to have to cook it from scratch with these bland ingredients.--------------[twitter]Overview of coding multitouch for the web: http://j.mp/wk87qC The building blocks still primitive, but they're there.[/twitter]

The browserhems you in

swipe

tap

Second thing. The browser itself gets in the way of your gestures.Because it has its own gestures.

A double-tap, a pinch, a tap-and-hold long touch.All have meaning to the browser itself.

When you build a web app, you’re running in an emulatorcalled the browser.

The browser has reserved these building-block gestures,you can’t use them yourself. Long tap. Pinch/zoom.

But, in terms of familiar conventions for gestures,gestures you can actually expect people to use and find,you’re basically limited to just two:[next]tap and swipe

Hopefully this will change as we start seeing more WebGLinterfaces on the web—makes the browser UI fade to background—saw some of it in Jen’s talk yesterday.

For now, though, really limited.

But wait, so if the browser sucks at this stuff,why are we even talking about this here?All the signs say: this is a conference for people who make websites.So what am I even doing here?

[twitter]Web gestures must sidestep built-in gestures in browser and OS. Apple’s OS gestures make it worse: http://j.mp/pqBXV3[/twitter]

http://bit.ly/browser-ninjaand you slash them by swiping through them.

So you can also detect multitouch gestures for the browser.

But again, it’s not exactly easy to go beyond simple taps and swipes.

That’s why, in terms of interaction design,we’re seeing the first generation of really innovativegestural interaction happening in apps.

So let’s look at a few examples.

Doing OK?

Adobe Proto

I want to talk about Adobe Proto.Likewise, new ways to do things, but here leveraging old familiar ways.

I want to talk about Adobe Proto.Likewise, new ways to do things, but here leveraging old familiar ways.

In this case, for drawing wireframes. The first phase of that is literally sketching. All gesture based. Check this out.

This takes a familiar physical model—sketching,applies it as-is to new format. And it’s cool.

[twitter]Adobe Proto: touch app offering quick wireframing w/gestures that borrow from sketch shorthand. http://j.mp/KNSwHs[/twitter]

http://bit.ly/ios-clearApps like Clear and Papershow how interfaces can be reimagined to create illusion of physicality.

Only buttons are the keyboard for typing.

Clear is a crummy to do list.NEEDS some buttons for more advanced features.Great proof of concept.Changes the way you approach the interface.

This is exactly the way we need to start moving,not to blow up all UI conventions outright,but to question them and ask:

In light of direct interaction, do I still need that button,or is there a better way to do it?If this data were a physical object, how would I stretch and change it?

More like an instrument that you play than tool that you use.Intent fluidly transferred to action.

Facebook Paper

I want to talk about an app called Touchup.

TouchUp

I want to talk about an app called Touchup.

As we embrace these new interfaces, also have to give up old abstractionsand mental models that we associate with desktop controls.

TouchUp: Draw filters or effects on photos.Brush these effects on, drawing them with your finger.

[next] What if you want a smaller brush?Traditionally you’d have a slider or some brush selector.But the thing is, you have a brush, and it doesn’t change size.

A setting to change my finger’s touch footprint to double or half actual sizewould just be confusing, hard to get my head around.

Instead of changing the brush size,you change the canvas size. Pinch to zoom in or out.

And then draw your next stroke.Finger always keeps its same physical size on the screen.It’s the canvas that changes size.

[slow]When you deal w/touch, have to rethink these familiar abstractions.

[twitter]In touch sketch app, brush sizes bad idea (finger doesn’t change size). Change canvas size instead. TouchUp: http://j.mp/qB5QTc[/twitter]

How do you render the physicality of a date range?Reimagine a date range as a physical object. How does that work?

All good?

What can youcount on?

Mentioned before that we have tap and swipe on the web as the only reliable gestures.

Not tons better in apps, to be honest, but we do have a set of foundational, primitive gestures that we can rely on now.

Let’s just run through those.

Tap, activate, show me more.This is the ask-the-question thing that drives everything.

Hover commentary.

Swipe. Almost always next/previous.That can mean scrolling, or it can mean flipping cards.But there are somewhat creative ways to use it, too:

To cross something out, for deletion.To slide a hidden panel, reveal actions below, or to reveal a drawer.We’ll see some examples of those more advanced interactions in a bit.

Double tap.

Zoom in and out.

Pinch and spread, of course, the more playful zoom in/out gesture.

Can also be used to close or open a view, and we’ll see examples of that coming up.

Tap and hold brings up contextual actions or a contextual menu. Treat this as the right click of touch screens. Not all users discover it. Often a shortcut to clicking through to the item to see what actions you can perform on it.

That covers standard, easily discoverable gestures.But what about the fancy stuff, the tricky, abstract gestures?

Well, there’s a problem there, isn’t there?

★ Buttons require cognitive and physical effort

★ Gestures = keyboard shortcuts★ Use the entire screen as a control★ Standards emerging

Tap, swipe, pinch/spread, long tap★ Model data as physical object

We’ve solved it 1000 times over.The color picker. More color pickers! A million styles of color picker.

All of these things let you get at any of 6 million colors,but it’s on you. Go ahead pick one. I dare you.

Paper went back to the source. How do we really choose colors?What’s the physical metaphor of picking, creating colors?

http://bit.ly/paper-mixerWent back and look at how you mix paint.That basic childhood, dead-simple way of getting the color you want.Choose a color and mix it in. Swirl it right in.

Took them a full year to get the math right on this.Crazy complicated to implement behind the scenes,but dead simple in practice.

Direct interaction and physicality lets you thinkin new and often simpler ways to interact with content.

Here, borrowing directly from real world.But even more interesting when you takephysical concepts and apply in new waysto natively digital elements.

[twitter]Paper for iPad cleverly reboots color picker for touch: http://j.mp/paper-mixer Took a year of hard math: http://j.mp/PS2ZYi[/twitter]

designing forDISCOVERABILITY

Finding WhatYou Can’t SeeFinding WhatYou Can’t See

How do you find out about these gestures?Gestures unlabeled. Invisible. Rely on visual clues or past experience.

Less a gesture resembles physical action, more difficult to find.More abstract gestures tend to go overlooked.

People will figure out stuff that works fromphysical or mouse-driven experience

Newcomers DO discover tapping twice zooms in.Double-click training from desktop computers kicks in.But NEVER find two-fingered single tap.

Train people by using conventions they already understand.One way to do that, as we’ll see, is creating realistic, physical interfaces.

But understand that with a little help,people will learn to work your interface sight unseen.We do this all the time.

We use interfaces daily that are essentially invisible, or at least in the dark.Too many of us can even hit that snooze button in our sleep.It’s muscle memory, like typing.

http://www.flickr.com/photos/37996588780@N01/346360988/

Most of us here no longer look at the keys when we type.Studies: ask professional typists to write down order of the keys,can’t do itMuscle memory, not part of conscious knowledge.

We do it without thinking.Access that subconscious knowledge faster than conscious.

The trouble with both of these examples:had to learn clock and keyboard before you could do this.Layout of the alarm clock before snooze in dark.Hunt-and-peck characters before touch type.

Nearly everything we know has to be taught, learned, observed.

So before we get to helping people find INVISIBLE gestures,I want to talk about the importance of visual cues.

[twitter]Muscle memory faster than visual memory, so physical UI (instruments, keyboards, touch UI) faster than visual once we learn it[/twitter]

We rely on cues in environment to help until we obtain mastery.

Cutting board for obsessive-compulsive chefsPrecise length and width of julienne vegetables for example.Means you don’t have to rely on memory, or muscle memory.It’s a cheat sheet, a reference.This is where we always start. Some kind of physical reference.

Surrounded by these little hints all the time,some a conscious set of tools or personal organization...but others are softer reminders, often social,and not entirely in our control as either designers or consumers.

[twitter]OCD cutting board ideal for perfectionist chefs: http://j.mp/IUK7jG (Good example of visual cues teaching muscle memory.)[/twitter]

http://www.flickr.com/photos/blackcountrymuseums/4385115536/

Don Norman’s new book Living with Complexity is a terrific read.And in it he talks about salt and pepper shakers.Which is salt, and which is pepper?[next] right is pepper? right is salt? who has no idea?[next] Don Norman: actually doesn’t matter which is correct.All that matters is what the person who fills them believes.

Arbitrary. Socially defined. Not evenly understood by all.[slow] Social signifiers only work if everyone knows them.Cultural understanding is messy.

So we have uncertainty. Forced to proceed with caution.Test shaker to see if salt or pepper. Slow down. Lose confidence.Our job is to give users confidence that they understand the system,move efficiently and confidently to their goal.Which should always be salty food as far as I’m concerned.

Cindy yesterday, compared us to hosts at a dinner party.A good host makes people feel at ease.As designers, we are the hosts filling the salt shakers.To say, “anyone who doesn’t know which one is salt is stupid” isn’t enough.Social understanding is not so neatly synchronized.

UI conventions are social constructions.We can’t give machines perfect intelligence about user expectations,but we can at least give them good manners.

[twitter]“Living with Complexity” by @jnd1er is a great read about taming complexity by minimizing complication: http://j.mp/oVuu0O[/twitter]

http://www.flickr.com/photos/ella_marie/3142159351/

To help prevent confusion among our guests.So we need to provide signals to prevent errors, confusion.Good design makes the problem go away so that these questions aren’t asked in the first place.

Labeling is one solution.

http://www.flickr.com/photos/smi23le/2420961722/

This is even better. No labels at all, just instant intuitive understanding. I know what it is, because I can see it.

The content itself is the label.Want salt? Grab the salt! No middle man involved. No processing of labels or signs.

Who needs a control when you have the content itself?

Touch interfaces allows direct interaction with content itself.Don’t touch a button, touch content you want to work with.

Photos app uses content as navigation.Information as interface.There’s no chrome here, just lots and lots of content.Tap the photo to see it.[next]

How can I let people interact with actual contentinstead of buttons,instead of gray interface controls?

This isn’t about change for the sake of change.It’s not gratutitously changing interaction.Because the interaction should be invisible.

Jon yesterday quoted the amazing UX designer...

The medium is the message

Jen mentioned this Marshall McLuhan quote.

In the 60s, he said:the medium is the message.

I think we may finally be moving to a place where...

The mediumis themessage

The message is the medium.

The content is the control.The information is the interface.

You can have lots of features and complex informationwithout having all the chrome.

But glass salt shakers and photo thumbnails are obvious visual cues.Evident calls to action.

What about more abstract gestures,pinch gestures we saw earlier or a 3-finger swipe?

This is where we often start.Instructions.

Make people read a manual before you get started.This is where you startwhen you use Popular Science app for the first time.

Slog through all the controls before you get to step one.

New Yorker does the same.LOTS of apps do this,asking you to read about how to work every bit of the app,before you even understand what it can do for you.

This is not only premature,but it also makes it feel more complicated than it needs to be.

Khoi Vinh, former design director for NY Times Digital,poked a little fun at this trend in iPad magazines.Here’s his cheat sheet for using a magazine.

Pages stacked on the right side display unread content.Pages stack on the LEFT display your reading history.

Pokes fun at just how overwrought this makes the experience.Instead of making it easier,up-front instruction manuals make it seem more complicated than it is.

http://www.subtraction.com/2011/03/28/an-illustration-for-stack-america

[twitter]Front-loaded instructions make apps feel HARDER to use. Send-up of iPad magazine apps by @khoi: http://j.mp/hxHHpM[/twitter]

Norwegian TV had a comedy sketch many years agoMedieval Help Desk.

Even if he COULD read the manual, he wouldn’t read it.Nobody reads the manual.

All of us have only incomplete knowledge of tools we use everyday.

Why don’t we read manuals? We’re impatient.We have a goal when we’re using a product or software,and instructions seem like a diversion from that goal,even when they could actually get us there faster.

Doesn’t mean you shouldn’t havecomplete reference for power users,but you should treat it as just that: a reference, not a primary learning tool.

Because nobody will use your manual that way.

Since we can’t get people to read, we often try show and tell instead.

Medieval Helpdesk: http://j.mp/rdlctL

[twitter]What if books required instructions and help-desk support? Medieval Helpdesk: http://j.mp/rdlctL[/twitter]

So most people have only partial understanding of their tools.

It’s also why most people think Monopolyis a long, tortuous game that never ends.

Because people don’t play by the rules.When people actually read the instructions and play by the letter,the game actually goes pretty quickly.

But god, this doesn’t exactly promise an evening of fun.

Why don’t we read manuals? We’re impatient.We have a goal when we’re using a product or software,and instructions seem like a diversion from that goal,even when they could actually get us there faster.

Doesn’t mean you shouldn’t havecomplete reference for power users,but you should treat it as just that: a reference, not a primary learning tool.

Because nobody will use your manual that way.

Since we can’t get people to read, we often try show and tell instead.

http://bit.ly/our-choiceMaybe using this guy.He wrote a book called Our Choice, and it’s been adapted into an ebook with a really smart, button-free interface. The interface largely self-explanatory, but Al took no chances and launches his app with this video.[watch]Well me, too, Al, because jesus, you really sucked the joy out of my first two minutes of this app experience.

Everybody does screencasts.But friends, nobody watches them except us geeks.

Still, you can see from watching this one,no buttons, no chrome, great interactions.

Complex actions will always require some instruction.And as I’ve said we have to give visual hintsto let people know how to work your app.

But instruction can come in many forms,not just a manual or a lecture,though we usually fall back on those solutions.

[twitter]Love the interactions in Our Choice app, but you have to slog through Al’s dreary screencast first: http://bit.ly/our-choice[/twitter]

Rise app describes gestures for setting alarms in the app.

This is the classic tutorial device.And it’s logical: show don’t tell.Better than a manual, at least it’s in context.Sort of.

Show. Don’t tell.

That’s classic of good storytelling. But we’re doing something more thantelling a story, doing something more than DESCRIBING.

We want people to actaully do this stuff.

Do. Don’t Read.

This is their wont anyway. They dont’ want to read or watch movies.They want to do.

And it turns out we learn best by doing anyway.So let’s take advantage of that.

Mailbox app. Commiting actions to muscle memory requires repetition of those tasks. The closer you can get people to the context of those tasks the better. In that way, the Mailbox app has a superior tutorial approach relative to Rise. In Mailbox, you actually interact with the tutorial to repeat the gestures it describes

So there are some ways that instructions up front CAN be useful.The closer to context the better.

It gets even better than that, though,and we’ll return to the best way to do tutorials in a moment.

Nature doesn’t have instructions

The best interfaces, of course, require no instruction.Nature itself doesn’t have labels,neither do most obvious and useful designs.

That’s not to say that nature is easy.Really difficult interface.Look at toddlers, see how hard it is.We’ve invested a lot of time in learning this interface.

So lean on that acquired knowledge in ways that don’t require instruction.If done right, realistic 3D interfaces invite touch.Not just “ooh, I want to touch that,”but hints about how thing works.

Texture & physicality give clues about what to touch.Apple emphasizes in its interface guidelines: Make it realistic.

Have to do more than throw some photoshop textures at it.Apple’s own apps show the hazards.

Calendar:ooh, a lovely datebook.

Very pretty, and of course...you naturally know how to use it.Just swipe to turn the page.

[next]

To turn the page, you have to useone of these tiny buttons at bottom.

[next]

You have to embrace your interface metaphor.If you’re going to make it look like a book,make it act like one.

Contacts: Tapping doesn’t turn pageSwipe deletes

Your interface metaphor suggests how to use the app.Here, the book metaphor is a confusing misdirection.Creates expectation that works like a book,but really through desktop-style buttons.

Love the oneyou're with

If you go this route of aping a physical object:EMBRACE THE METAPHORThink hard about new interaction opportunities it proposes.

In user testing I see this all the time:If it looks like a physical object,people will try to interact with it like one.

Have to follow through.Don’t make me tap buttons if interface suggests otherwise.

Meanwhile, many magazine apps are TOO literal.Very true to original print artifact.Little more than PDFs. Clear exactly how to useBut difficult to find table of contents: no random access

As you embrace these realistic interfaces,don’t forget to enable what digital media can do best.Unshackling us from linear reading experiences.

Here’s an example of tweaking the real worldto digital advantage. Met these guys last week in Aus.Sydney Morning Herald.

It looks like a newspaper, and you swipe through pages.Page indicator to show how.But even better, can quickly scan each and every article in today’s edition.

Calendar and contacts: designers followingtheir metaphor half-heartedly in interactionMags: too literally and they miss interaction opportunities

This is understandable: Tablets in particular a weird hybrid.

Not a web, phone, desktop, paper/physical interface.Yet suggests elements of all.

Great opportunities, lots of tradition to draw from.But have to be careful about mixed metaphors and misdirection.

[twitter]Sydney Morning Herald app has 2 modes: newspaper browsing & direct link to content. Best of physical/digital. http://j.mp/Naxsi7[/twitter]

Model your apps to help people learnsame way they learn in the real world:Right from our earliest days,we rely on physical hints about how something worksand then verify it through the feedback the thing gives us.

Watch how toddlers use iPad.Amazing how quickly they get it: direct manipulation of content.Tap content directly, drag content across screen, nudge views away.They won’t get your multi-level menu navigation,but they’ll get any physical metaphor.

Follow the toddlers. They’re better at this than you are.Toddlers haven’t been poisoned by 30 years of desktop interfaces like we have.Find a three-year old and make her your beta tester.Think: would my toddler figure out this interaction?

Play more video games

This is your homework assignment.Go to work next week, settle into your desk, and fire up some games.Tell your boss you’re doing research, because you are.

Video games are great at teaching unfamiliar controls, unfamiliar skills.

In many games, you start without knowing anything:you don’t even know your goal, let alone what you can do,what abilities you might have, how to control them?

Sound familiar? That’s the same problem we have with teaching touch.So how do they do it?

[twitter]Want to be a better interaction designer? Play more video games.[/twitter]

Coaching

Leveling up

Power ups

1

2

3

Among other things,video games use these 3 tools to teach uswith visuals and experiences.

Every modern theory of learning emphasizesimportance of active participation,of active discovery,supplemented by coaching and mentoring.

Coaching

Leveling up

Power ups

1

2

3

Coaching involves simple demonstrations,prompts to tell you what to do.

This is the game riding along with you,pointing things out as you go. We learn by doing. We learn best in the moment.Telling people HOW not nearly as effective as coaching them thru.

Teaching HOW to do things is hard.Consider teaching music, consider teaching a tennis serve.Best taught by demonstration and learned by practice...

So help me in the moment. Show me. That’s what games do.

The arrows overlaid on the hero in the first screenof iPad game Dead Space show you how to move.

Simple temporary overlays that tell you what to doin the moment can be super helpful.

The walkthrough

Games are great at walkthroughs.They’re an improvement on the straight tutorial we saw with Mailbox and similar.

Tutorials are effective at teaching controls,but they also feel like stilted lessons,a series of hoops to hop through before you can use it for real.

By contrast, the walkthrough teaches by taking you through the actual app,with actual content.In most cases, walkthroughs restrict you to a rigid path of narrowlydefined features.

It’s like training wheels: you’re moving and going on your own,but with a few constraints to keep you from falling.

App: Noted.

So you’re actually USING the app while it walks you through. Now this is also a set of steps you have to follow. Can’t really do anything on your own.

So another approach leans on sending you notes as you encounter them.

Facebook Paper. Your actual content, you can use it as you like. But when you hit the context, it tells you what to do.

Same principle for these little tutorial boxes in many websites.

Gmail uses them in getting-started tours of an app,as well as to call out a new feature.Facebook does something similar.

Your app can, too.

Some of you might be squirming a little bit,because this might seem a little bit like a certain interface flop.

Are you trying to write a letter?

A flop named Clippy.

Clippy was never helpful.He was only a distraction,popping up at the most inconvenient moments.

Are you trying to write a letter?

The problem wasn’t so much Clippy concept,as the inane content that he had.Only offered to do the dopiest things.

Done right, though, assistant featurelike you see in Gmail or Facebook or in so many gamescan be helpful to give information at appropriate times.

It can be subtle and respectful.

But you can be more subtle, too, and animation can be useful coaching.

USA Today animation.

Coaching

Leveling up

Power ups

1

2

3

An important component of coaching:you don’t teach everything all at once.

We learn best by getting it in doses.Build on the basics and then reveal more as you go.Teaching a kid football, you don’t hand him the 200-page rulebook.You start with the basics and gradually build,teaching skills only as they’re needed.

Ease players into the app.Introduce one element at a time.Encourage people to master the controls and features a little at a time.

Best time to learn a skill is right at the moment when you discover it’s needed.That’s when we’re most motivated, most engaged.

Like, for example, when you encounter a gigantic and hugely scary guy with a big-ass sword.

This is Inifinity Blade for iPad.Crazy sophisticated combat system,but easy to learn by teaching you one step at a time,introducing challenges that are specifically designed to test those skills.

The game pauses itself right here, freezes it,when you’re about to get your ass kicked,to tell you, hey, hold this shield to block.

When you do, the action starts up again,and whattya know, you’ve blocked.Your first try is a guaranteed success.Now you’re ready to try it yourself.

In games, levels that introduce a new featureshould focus on teaching that feature.Likewise, when people first encounter a feature is when you should offer coaching.

In some very important cases,should interrupt people to force themto demonstrate they’ve learned before continuing.That’s whole principle of a fire drill. Show me you know how.

That’s what Infinity Blade does over and over again,pausing the action at incredibly convenient timesto teach you how to use a move to beat your opponent.

It waits until you demonstrate you understand.Again, first time is always a success.

OS X Lion does this when it first installs,explaining you must swipe up to scroll down,instead of the old way. It MAKES you do it.Actually scroll before you can see the continue button.

Boom, you’ve completed the first level of the OS X Lion game.

Coaching

Leveling up

Power ups

1

2

3

Whole concept of levels is about evolving from beginner to expert,and expertise is where the fancy moves come in... the power ups.

In video games, power ups like the super mushroomsin super mario brothers give you some kind of superpower.They turbo-boost your game, giving you shortcuts or some other advantage.

Power ups are the keyboard shortcuts of video games,usable by anyone but especially efficient in the hands of an expert.

But it’s also a reward. It’s an achievement.There’s a thrill to that. A thrill to discovering new features.This is true of the tools we use every day.

And that’s what gestures are, too.

Shortcut!Swipe up fromMe button for

direct messages.

So you have to make thing discoverable.

The need for this kind of help again points up the fact thatwe don’t yet have gesture conventions.

[twitter]My thoughts on gestures and the need for contextual help in Twitter for iPhone: http://j.mp/vZpRur[/twitter]

Be generous

That means that it’s more important than ever to share.Pick up on what Eric said.This stuff is hard. Don’t go it alone. We have to talk about it.

Gestures are invisible and have to be learned.We’ll stagger at this.

Don’t assume anyone else has nailed it or tested it,or thought about it.Everyone’s flying by seat of their pants right now.This is very early, and it’s dangerous to lock in on half-baked conventions.

Experiment. Ask questions. Find out if this is solid.Need to help each other, we need to have conversations.

[slow] This is a time to be generous.

designing forTHE FUTURE

Friends, I propose an expedition.Where no geek has gone before.

Exploring the hazy edges of the technology universefor emerging tech trends changing the way weinteract with devices, with information.

We’re looking to the future to see how we shouldthink about our work now.

You are here

First, though, I want to start our mission with where we are now.[next]

What it is that this new era of computing gives to us that we haven’t had before?I’d say the big, big shift...

Sensors are the

superpowers.

SENSORSgive us

SUPERPOWERS

...is sensors.

It’s the sensors that give rich context and insight to that info and tasks.

GPScameramicrophonetouchlight detectorproximity detector,accelerometercompassa GYROSCOPE.

Mobile is often considered companion, lite.Wrong. The question is not how to strip down an experience. Not less, more.These devices can do more than the desktop.

Often we use sensors for immediate proximity, maps.Where to eat dinner nearby. When the next train at my station is leaving.But I encourage you to think beyond just geography with these sensors.Not just what’s nearby but what’s in front of you.

Shazam showed us we can use audio to do some magical parsing of what’s happening around us.

Use sensors to parse, interpret, and make smart inferences about our immediate environment.That’s great.

[twitter]Mobile isn’t a lite version. Don’t ask how to do less w/mobile, ask how to do MORE. Sensors let mobile do more than desktop.[/twitter]

AugmentedReality

To add new layers of understanding and insight,by adding new visuals to what we can see with our puny human eyes and ears.

I think we have to be careful to use it correctly,because it’s something that can quickly become a gimmick.A solution in search of a problem.

There are a lot of bad implementations of augmented reality.Something that looks cool, but that you wouldn’t actually find useful day to day.

There are a few areas, though, that have been quite compelling.The first is games.

[twitter]Augmented reality is almost always a lame gimmick. But a few compelling uses are emerging: games and translation.[/twitter]

Skinvaders

Skinvaders is a game that uses the front-facing camerato turn your face into a game board.

Where, um, aliens invade your skin.

[twitter]Skinvaders augmented reality game has aliens that, um, invade your skin: http://bit.ly/skinvaders[/twitter]

Ikea

The idea of games is to introduce whimsical what-if objects into your environment.

Ikea doubles down to introduce whimsical what-if furniture.

Scan stuff you like in the catalog, and then drop the catalog in your homewhere you’d like to see how it looks.

Next best thing would then be to 3d-print the furniture from there.

Layar

But with Ikea, the launch point is the catalog, the paper product.Let’s keep on with that theme.

Layar began as an augmented reality browser,reinvented itself as a product for editorial,what they call interactive print.

The idea that any layout is unique, is itself a QR code,that can be read like a design fingerprint.

Take a look:Use to insert video into static pages.Or add traditional links to download apps.Or buy products.

You can build your own layar interactionsthrough their website with something calledLayar Creator.

IF YOUR BUSINESS HAS PAPER ANYTHINGCatalogs, insurance forms, whateverThis gives digital presence to these physical objects.

But more important, it’s fast.It works like barcode or qr code.Scan the thing, go to its digital representation.It speeds input.

Let’s look at another example.

word lens

Word lens is an augmented reality app that uses computer visionand optical character recognition to translate text in real timefrom one language to another. [next]

It’s a boon for travelers. You’ll never order tripe by mistake again.No internet connection required.Even keeps the same general font style and color.

Really what this is doing is saving input.

Word lens is an augmented reality app that uses computer visionand optical character recognition to translate text in real timefrom one language to another. [next]

It’s a boon for travelers. You’ll never order tripe by mistake again.No internet connection required.Even keeps the same general font style and color.

Really what this is doing is saving input.

Input

MaximizeOutput

Minimize

Seeing this, too, in Google’s explorations in predictive and assistive technologies.

For many, Google has everything about you,your email, your calendar, your friends, your map searches, your location

They use that info to have a pretty good idea of what you’ll want to do at any moment in time.

No UI required at all. It’s all in the data.

Let’s continue with this idea of no UI,or at least no screen UI as we know it.

Table Drum

We’re used to augmented reality as being entirely visual,camera based. Table Drum takes a different approach.

Table Drum is a drum machine app. There are lots of those.What makes Table Drum different is that it doesn’t force you to usethe screen. Pushes its interface into the world around you,so that you can actually “play” the table in front of you.

[next]

The developers call it augmented audio.The software uses its sensors to push beyond the screen.Every object in front of you is suddenly a sensor, an input.

Do you see the commonalities between these apps?Most of these examples are replacements for traditional input.We think of iOS devices as touch interfaces, and that’s awesome. Lots of possibility.

But crafty use of onboard sensors means you don’t have to interact directly at all.

That’s the promise of natural gesture of course.You know this stuff from Kinect where the interface is your body itself.Move your body to interact with a screen.

And Microsoft Research is demo’ing displays and laptops with miniature Kinect cameras built right in.

Anna mentioned both Kinect and Leap Motion. Let’s look at that...

[twitter]Table Drum uses sensors to shift interface off screen, into environment. Whole desk is drumset. http://bit.ly/tabledrum[/twitter]

We’re used to augmented reality as being entirely visual,camera based. Table Drum takes a different approach.

Table Drum is a drum machine app. There are lots of those.What makes Table Drum different is that it doesn’t force you to usethe screen. Pushes its interface into the world around you,so that you can actually “play” the table in front of you.

[next]

The developers call it augmented audio.The software uses its sensors to push beyond the screen.Every object in front of you is suddenly a sensor, an input.

Do you see the commonalities between these apps?Most of these examples are replacements for traditional input.We think of iOS devices as touch interfaces, and that’s awesome. Lots of possibility.

But crafty use of onboard sensors means you don’t have to interact directly at all.

That’s the promise of natural gesture of course.You know this stuff from Kinect where the interface is your body itself.Move your body to interact with a screen.

And Microsoft Research is demo’ing displays and laptops with miniature Kinect cameras built right in.

Anna mentioned both Kinect and Leap Motion. Let’s look at that...

[twitter]Table Drum uses sensors to shift interface off screen, into environment. Whole desk is drumset. http://bit.ly/tabledrum[/twitter]

touch interface

no touchIs sometimes

at all

Best the

But I’m realizing...

How can we save the user from tapping into the screen at all?That’s the point of barcodes, of computer vision recognition,of speech recognition.

Speech.One of the powerful things about sensors is that we cancommunicate with machines in new ways. Often in more human ways.

Touch is just the first, it’s finally mature.But obviously speech and natural gesture are ready to pop, too.Facial recognition.

All of this stuff is still a little unreliable.Siri is in beta.Kinect great for games, but wouldn’t want to run a nuclear power plant with it.

Design for

sensors

NotJust

Screens

Men, Women & Childrenbit.ly/mwc-trailer

Even when we’re surrounded by people,we’re heads down in screens.It’s become a primary way that we interact with people and our environment.

We’re heads down on our phones when we’re at the dinner table,or, ahem, in bed.

This is the trailer from Jason Reitman’s new film,Men Women & Childrenwhich paints this rather dreary social picture.

I’m not sure it’s this bad. But I do think the screen is to blame.

I would say that it’s up to us to change the trajectory of that trend.Because at the moment, we’re building stuff like this...

http://www.coroflot.com/Lylover/Surf-Chair

The Surf Chair.

That’s not going to help.But why stop there, when we can start much, much earlier?

We’re putting screens on everything.Our poor kids don’t get a break.Here’s a little parenting revolution to help potty train kids.

The 2-in-1 potty for iPadhttp://www.ctadigital.com/item.asp?item=3016

CustomSensors

Stuff gets really interesting as engineers create custom sensorsfor the interfaces. Let devices talk to any arbitrary object.

It all starts with this sensor that you put on your inhaler.It talks to your phone via bluetooth and says, hey I just took a puff.The phone adds GPS information and relays time/location info to the server.

And you can add that event to your phone, optionally adding info about the symptoms and triggers for the asthma.

The system can then tell you whether or not your asthma is controlled — more puffs mean less control.

And give some general insight into the conditions that seem to cause your asthma.

You can see where you tend to have attacks,which begins to get at time and trigger events.

Aggregate the info,epidemiologists can see what’s happening in a community.

Hey, tons of people have an attack 2 hours after passing through this area.So it’s a way to start understanding the causes of asthma in any community.

Custom sensors plus big data let us find patterns that can help not onlyindividuals, but whole societies.

Healthcare field especially innovative in turning phones and tablets intoinexpensive medical devices.

http://www.popsci.com/bown/2012/innovator/proteus-digital-health-feedback-systemProteus Digital Health Feedback SystemPill that doubles as a radio, so it can track whether you take it.

Sensor itself about the size of a grain of sand.Same stuff you find in a vitamin.Copper and magnesium hit your gastric acid, turning this thing into a battery.Works like a potato battery

Transmits a snippet of code to a patch you wear on your stomach.,which relays to your phone or tablet via bluetooth.

Pill’s serial number, manufacturer, and ingredients.saves that data to the cloud.

A phone or tablet that knows what’s going on inside your body.

[twitter]Sensors everywhere, even in your body. A pill that tells your smartphone when it’s taken: http://j.mp/Y1Zr5m[/twitter]

This is some groovy Star Trek stuff here!Like I said, we are living in a world of science fiction.----Advanced pulmonary disease.Sensor inside the artery near the heart,detects blood pressure changes.

Touch a sensor and it downloads the info THROUGH THE SKIN.Shows the patient some information and tips,relays the data to the doctor.

----

As sensors become more advanced,there’s more stuff we can do with our DATA,with our content.

----

Sensors are becoming so cheap.So trivially inexpensive to turn physical objects into digital broadcasters,we’re seeing experiments in the most unusual places.

So to turn from the sublime to the ridiculous...

So that’s amazing.But taken to extremes, it can also get a little silly.Here’s a Mickey Mouse idea, honest to god,that Disney Research came up with last month.It’s called Botanicus Interacticus.

PLANT UI! IMAGINE THE POSSIBILITIES!

[twitter]Plant UX! Botanicus Interacticus is multitouch plant controller from Disney Research: http://bit.ly/botanicus[/twitter]

Photo by Martin Abegglenflickr.com/photos/twicepix/1397862091/

Farmers in Switzerland are trialing cow sensors.These things detect when the cow is in heat[play]

...and then send the farmer a text.And these texts can go out in French, German, or Italian.Cow love speaks all languages.

Forget internet of things, I want my internet of bovines.We have cows texting when they’re in heat. That’s basically how I use SMS, too.

[twitter]Sensors everywhere: even cows. Swiss cows send texts when they’re in heat. http://j.mp/Rq5o8Z[/twitter]

Becomes physical

PHYSICALBecomes digital

Digital

So... we’ve been focusing on making digital physical,etching digital interfaces on these slabs we carry out into the world.

At the same time, the phyiscal has become more digital.We have this growing variety of sensors in the world around ussharing data in all kinds of ways with all kinds of devices.

Any physical object can communicate informationor even control our digital interfaces.

Sensors

Mirroring

So I would say that the big marker of where we are noware personal sensors.But your device is not only a sensor for input...but a broadcaster.

So if we embark on our mission with a quick hop...

We get to mirroring. For the most part, that simply means screen sharing.

Airplaybitly.com/airplay-ad

That’s the whole idea behind Airplay.Mirror your screen via Apple TV to your television set.To share photos, to share videos.

That begins to make it social.It shares it display with other devices -- and, ultimately, with other people.

So, it’s not just a sensor.Your device broadcasts content to dumb devices.It becomes the sensor and receiver for those dumb devices, like TV.

Samsung takes another approach.

Instead of mirroring, of having one smart device broadcast to a TV,they say: let’s just make every device “smart”give them independent operating systems.

Who needs to mirror, they say, when everything can be a smart device.

So you have things like the Samsung Smart TV.[watch]

A whole internet-enabled operating system, with speech and gesture.Xbox built right into the tv.

This isn’t a new concept.“Everything’s going to be smart!”

For some reason, we especially hear about that future in the kitchen.Futurists obsessed with idea of smart refrigerators and smart toasters.

At CES this year, Samsung introduced its T9000 refrigerator with LCD and Evernote integration.

Maybe there is a place for smart appliances.But making them all browsers or evernote clients or twitter clients is the wrong way.

First of all, expensive and out of sync with the life cycle of these devices.

But more important, they miss the point:Connectivity isn’t always about browsing and tweeting.

It’s about making an appliance a better appliance.WHAT JOB DO YOU HIRE IT TO DO?Microwave ovens.

So going back to the refrigerator, how do we think about that?

http://www.engadget.com/2013/01/07/samsung-debuts-t9000-refrigerator-with-lcd-and-evernote-integrat/

Here’s one example.

Eggminder. General Electric and a company called Quirky.

Its whole job is to keep track of how many eggs are in the fridge.

So when you’re at the store you can see how many eggs you’ve got.

HOW LAZY CAN YOU BE.

Well that’s one of our jobs as designers.Let people be as lazy as possible.Trivially inexpensive to add these things to objects.

How can we make tiny incremental improvements?

I DON’T WANT MORE OPERATING SYSTEMS IN MY LIFE.I already have too many.

Too many patterns to learn, too many technologies, too many contexts.I’m not sure that more smart devices is actually what I want.

So how do I get from here...

How do I make peace with all these devices and screens?

Let’s push further out in our mission.

Remote Control

Part of maintaining sanity is limiting the number of smart interfaces we have to deal with.

Have a handful of smart devices that can control all the dumb devices in our lives.

Drive everything, for example, with my phone or tablet.

[twitter]I don’t want smart TV, smart toaster, smart fridge, each w/ own OS. Let smart device I already have (phone/tablet) run whole show.[/twitter]

Metalstorm:Wingman

Apple’s Airplay lets you do that, too, though you see it far more rarely. Here’s one example. [next]Metalstorm Wingman uses the iPad to fly your plane on the tv.

Beyond mirroring, obviously.This is a generic device acting like a purpose-built controller or remote,working with a dumb device, the TV.

This is using Apple’s proprietary system, with Apple TV.

Microsoft likewise has Smartglass,to control your xbox or get related content on your phone or tablet.

Apple’s Airplay lets you do that, too, though you see it far more rarely. Here’s one example. [next]Metalstorm Wingman uses the iPad to fly your plane on the tv.

Beyond mirroring, obviously.This is a generic device acting like a purpose-built controller or remote,working with a dumb device, the TV.

This is using Apple’s proprietary system, with Apple TV.

Microsoft likewise has Smartglass,to control your xbox or get related content on your phone or tablet.

Chromecast.This little guy plugs into your TV.When you watch video in the Chrome browser on any device—phone, tablet, laptop, whatever—tap a button and push the video over to the TV.

SEEMS like airplay, but it’s really remote control,telling the tv to grab a video stream directly from the web,so it doesn’t actually stream from the original device.

Remote control.

Internet-enabled remote control crock-pot!

http://www.crock-pot.com/slow-cookers/coming-soon!-crock-pot®-smart-slow-cooker-enabled-with-wemo™/SCCPWM600-V1.html

bleep bleep bloop

oh hai!

So we’re talking about devices talking to one another, directly.But what if they literally talk to each other. Like with sound.

So one computer says this......and the other hears and grabs the data and responds.

They used to do this all the time over the phone.Just call each other up.

But what if browsers could do this. Talk to each other like this.Well, it turns out they can.

Boris Smus @borismus created this library called sonicnet.js.It uses the web audio API to make browsers make and listen for audio.In this demo, he’s associated a sound with different icons,so you can broadcast an icon from one web browser to another—no server required.[play]This also works at super high frequency.So only the browsers and your dog can hear it.

That last example sends a string of characters, an authentication token,so you can use this to pair two devices via a third-party server.

Very similar to what a company called SlickLogin does.SlickLogin acquired by Google in late 2013.Your phone broadcasts a unique ultrasound signal.Put it down next to your computer. And boom, you’re logged in.

----http://smus.com/ultrasonic-networking/http://www.engadget.com/2014/02/16/google-acquires-slicklogin-sound-passwords/

Racer is a multi-screen racing game that runs on the web,in the Chrome browser.

SHARING CONTROL across devices.Not just content, but an experience.And doing it in physical space, not just virtual space.The spatial relationship is what makes the whole experience so special.

The technology that’s making this data transfer possible is WebSockets.A server in the middle sets up a persistent data stream betweenall the gadgets and drives the show.

But you don’t actually need that server at all.If we want gadgets to be truly good communicators,maybe we can let them talk directly.

And that’s where WebRTC comes in.

http://j.mp/racer-chrome

Chess + WebRTCbitly.com/webrtc-chess

Here’s a game that’s built with WebRTC to let players play chesswith each other—in the browser—via video chat.

The video chat isn’t happening with a server.That’s direct peer-to-peer video chat, entirely via JavaScript.A JavaScript videochat client.WHO NEEDS SKYPE WHEN YOU HAVE JAVASCRIPT?

Chrome, Firefox, Opera—exchange video, audio, or any arbitrary data directly with each other.Peer to peer, no web server needed.

WebRTC used mostly for video chat at the moment,but it can swap all kinds of data, so it can be used for:remote control,peer to peer file sharing,multiplayer gaming,anything that needs a persistent data stream between browsers.

WebSockets without the server in the middle.

But in any case, using WebSockets or WebRTC,we can exchange not only content but control. Action.

http://www.youtube.com/watch?v=KWuSnwD2gJ8http://www.jocly.com/node/339http://bit.ly/webrtc-chess

Sensors

Mirroring

Remote Control

So that’s pretty much the state of the system as we know it.This is where mainstream tech is pushing at the frontier.

In all of these cases, though, single smart device controlling the display.Either its own display or a remote display on a dumb device.Broadcast.

But let’s push that and go further out in our expedition.[next]Here’s where things get interesting...An important element of the near future will be more ambiguous control.

Shared control among devices.Primary shifting from one device to another depending on your context.

We know this concept already from the good old car phone.

[twitter]In device-to-device comms, remote control beats mirroring. But SHARED control that migrates among devices is even better.[/twitter]

Migratinginterfaces

So that’s pretty much the state of the system as we know it.This is where mainstream tech is pushing at the frontier.

In all of these cases, though, single smart device controlling the display.Either its own display or a remote display on a dumb device.Broadcast.

But let’s push that and go further out in our expedition.[next]Here’s where things get interesting...An important element of the near future will be more ambiguous control.

Shared control among devices.Primary shifting from one device to another depending on your context.

We know this concept already from the good old car phone.

[twitter]In device-to-device comms, remote control beats mirroring. But SHARED control that migrates among devices is even better.[/twitter]

Say you’ve got a car with a bluetooth phone system so that when you get a call on your mobile...

It comes through over your phone’s speakers.Your CAR starts ringing.You answer without taking your eyes from the road.The car itself is your interface.And you continue your conversation.

Until you park.

Migratinginterface

And then you grab the phone, mid conversation, turn off the car and continue your call, seamlessly on the phone itself.

Do you see what happened there? [next]

The interface itself shifted from one device to another.

The phone is still driving all the logic throughout,but interaction happened elsewhere while you were driving.

The car had over control of the call, even though it’s coming through the phone.This invisible shift of control from one device to another.

Surprisingly few experiments along those lines in the devices we carry now,but there are a few of these social hardware interfaces...

[twitter]Migrating UI: In paired car phone, UI shifts from phone to car (the car rings!), back to phone. Whoa.[/twitter]

Scrabblebitly.com/scrabble-4-ipad

Like this Scrabble game that came out a year or two ago.

Here, two iPad’s sharing control over the single game.

And, perhaps more interesting, two iPhones holding your letter tiles and then playing them to the iPad board in the middle.

And there you have it: the world’s most expensive board game.

[twitter]Scrabble with iPad and two iPhones. The world’s most expensive board game. http://j.mp/OISaY3[/twitter]

Corning is a glass company.Gorilla glass.

Obviously very invested in the success of screens everywhere,provided those screens are made of glass.

Concept video: A Day Made of Glass. They wish.

But pay attention, because there’s an interesting concept behind this.

Let’s start with this girl using a tablet in her bedroom.

[twitter]Corning’s “Day Made of Glass” shows plausible near future: lots of dumb UI sharing control w/smart tablet: http://j.mp/day-glass[/twitter]

Oh, is that all?

OH... is that all that’s missing?Just the operating system. And apps that seamlessly scale and transfer.

That’s a tall order, of course.But wow, the hardware is here.Now we just have to figure out how to code for it.

This is more than just mirroring, if you’ll forgive the pun.This is the tablet giving its interface control to the mirror.

I can have just one smart CPU with all my apps and content with me at all times,and then just call up a screen wherever I am.

What we’re talking about here is smart devices talking to each other,often with one core central device—our phone or tablet—running the show.

That means we have to design the spaces between our gadgets.

The rumor mill is going full-tilt that Apple will release an iOS watch this year.

How does your website or app look on THIS?

And even this seems easy in comparison to other stuff that’s coming at us.We’ve gotten pretty good over the past few years withfitting our content and messaging into different squares and rectangles.

Phones, tablets, etc.

But then...

The watch feels independent. You tap and swipe through the thing as if it were smart.But like so many other of these new smart watches, it’s actually fairly dumb.

The logic and the smarts and the sensors live in the phone.You have to have the phone nearby to make it go.And that’s reasonable at least for where the tech is now.Let your most personal smart gadget—the phone—drive the logic behind nearby dumb devices.

Gesture prototypesbitly.com/proto-gestures

Three years ago, a few design students put together a videoof paper prototypes for how smart devicesmight interact. And so they looked at what screen-based gestures might evolve.

I edited it down and thought you might find it fun.--That’s pretty charming, right, but seems kind of far-fetched.Shaking content from one to another, just sliding stuff from one screen to another?

Actually, we’ve already got it.

http://vimeo.com/7055121Jenny Redd, Kenny Hopper, Nicholas RiddleCalifornia College of the Arts, 2009

[twitter]Fun touch-gesture concepts for moving data from device to device: http://bit.ly/proto-gestures[/twitter]

Sifteo

The game runs on your PC.Connects to these guys via bluetooth.They are the interface for the game that runs on the PC.

They can detect each other’s proximity.They have accelerometers: tip and tilt content inside them,pour it from one to another.

It’s a complete little ecosystem, made for games.

Also, the little cubes download the software they need only when they need it.This is also an element that may be part of the future.

Machines that are smart enough to grab software when they need it.

And that kind of basic, almost reflexive intelligence,brings us to the next stop as we move farther out into the future of computing.

[twitter]Sifteo an example of simple social devices: toys that detect each other’s presence and interact, share. http://bit.ly/sifteo-cubes[/twitter]

Misfit Shine

But here’s a product that tries to introduce a similar social whimsy.

Shine is a product from a company called Misfit Wearables.It’s a fitness tracker like Nike Fuelband or Fitbit, but it LOOKS better.Most wearables aren’t very wearable.This one’s different, an elegant metal disc.Wear it like a pendant or a watch.

And watch how you sync it.[Video]Just put it right on the screen and the phone seems to absorb the data.Like it’s the physical touch that does it.

In fact, it’s a piece of clever misdirection.

On the web, we’re starting to get migrating control, too:Chrome RacerWebRTC

THAWbitly.com/mit-thaw

THAW is a project from MIT’s Tangible Media Group to make phones and laptopsmore friendly.[start]The phone can now interact with another screen, moving content around.

Or copying content over to the phone via mouse and cursor.

Or even acting as a second-screen interactive control.

See what I mean? Sharing not only content but action.And INTERaction.

Yeah. Eat your heart out, Zelda.

http://tangible.media.mit.edu/project/thaw/

Apple Continuitybit.ly/ios-continuity

Browsing the web on your iPad, the lock screen on your phone lets you tap and get that page right there in the browser.

Android and Chrome have had something similar with easy syncing of browser tabs.

The new wrinkle that Apple is banking on here is proximity. The computer knows you’re on another device right nearby working on an app that the computer also has. Hey, I can take that over for you. One click and boom.

Or you’re writing an email on your laptop, send it right to your phone and continue tapping away. Or vice versa, from phone to laptop.

Samsung just last week (Nov 2014) said they were getting into the act with Flow.

Very similar to Continuity but for Samsung’s build of Android.Let’s take a look.

When you’re browsing a website on the phone,you can hit the Flow icon, choose a device,and send that page over to the device.

Works with other types of activities, too.If you have a video chat on your tablet, send it to your phone to take it to go.

Samsung has lots of devices, of course, so they wantyour smart TV to be able to show you when you get a call.

Or send a map from your phone over to your Samsung Gear watch.

So solving a similar problem, moving SCREENS from one device to another, midtask.

That’s Samsung. The official build of Android is rumored to soon launch something called Copresence to exchange info between devices.Here’s a leaked marketing image from that.ADORBZ.

Rumored to build on Bump app which they acquired.

All the operating systems laying infrastructure,the pipes to send not only content but action/task/behavior.

I call this infrastructure, because I don’t think the interaction is very polished yet.I think that will change, and in a bit,I’ll look at some examples of how we *should* be designing.

As we’ll discuss I think there’s much more interesting interactions to explore.

In the meantime, that’s devices working together, SOME hint of exchange and interplay.

These are all operating-system changes. But friends, this is a web conference, right?What’s with all these native apps?

I’ve been working on some prototypes to do thatwith my buddy and studiomate Larry Legend.

We hacked an app and server together in a few afternoonsand made this video to show you.

We call it Happy Together.

When the phone and desktop are happy together,Larry’s happy.

In case you missed it, let’s see the instant replay.

http://vimeo.com/globalmoxie/happy-together

Physical context matters.Just like Luke was talking about where you’re sitting affects how much info you can take in.

The same goes for interaction. Physical context and mindset should inform how interactions should work.

What are the physical and psychological contexts for how you’re experiencing this media?

This is my friend Aral Balkan.Overnight hackathon.Content sharing hack with a kinect, a mac, and an iphone.

In these last few examples, we seepartner devices, equal peers, each with a role.

Often that’s what we’ll be looking for. Easy ways to swap info between devices.

It actually was pretty easy with Palm devices in the late 90s, just beam contacts and meetings to another device. Somehow got a bit harder, but I’m sure we’ll fix that.

But when we get it right, it feels magical.

How else might we do it?

[twitter]My pal @aral hacked together Kinect, Mac, TV, phone for amazing demo of moving data among devices http://bit.ly/grab-magic[/twitter]

Gesture+

speech=

Magic

This is the world we’re designing for. Spells.Working at a distance to transform information and,with the internet of things...

objects.

So... Pushing sensors lets us push our interfaces off the screenand into the world around us.

That’s just the onboard sensors...

[twitter]Don’t design for inputs like touch, gesture, speech in silos, but in combos. Speech plus gesture: spells.[twitter]

LEAP MOTION

This kind of stuff is called natural gesture.It’s what Kinect is made of. And this: Leap Motion.

Like a Kinect but for individual computers instead of a whole room.

It’s this little $80 gizmo that you put in front of your computer,and boom -- your app is the minority report.

It’s also getting built into devices.Remember Luke talking about gadgetswith all kinds of different input modes built into combination.

Ringlogbar.jp/ring

http://logbar.jp/ring/

Ring is a new gizmo that you wear on your fingerto allow all kinds of gesture control and even handwriting.

So we’re developing this ability to gesture at objectsin the air to make them do our bidding.

Which, among other things, means we can use our eyes to other stuff.Like drive.

Prototype from NYT.

Here’s a prototype of using a Leap Motion with an architectural model.

How can we use sensors and new input types to interact directlywith the physical world and give digital objects a physical presence?

Passiveinterfaces

Passive interfaces.

What I mean by this is devices that just do their work and talk to one another without even needing us to intervene.

If devices are already smart enough to talk to each other, share control, then they can also start doing that on their own.

We’re incidental to their behavior. We’re just the legs that bring the devices into proximity.

The Nest thermostat.Proximity sensors to know when you’re home.Humidity sensor.Temperature.Wifi.Internet connection to get the weather outside.

Talks to apps and website.Remote control and reporting.

And it’s a THERMOSTAT.A relatively dumb device but wow, fully loaded, and in constant communication.

Nest’s new smoke detector, called Protect.

Photoelectric smoke sensorCarbon monoxide sensorHeat sensorThree activity sensorsAmbient light sensorHumidity sensorWifi connection.Talks to apps and website.Remote control and reporting.

Tells you what room the smoke is in.

And it’s a SMOKE DETECTOR.A relatively dumb device but wow, fully loaded, and in constant communication.

The risk is creating an overly cutesy,hyper-programmed world of Beauty and the Beast.Objects come to life, they have personalities.

This kind of anthropomorphic design is the third rail of interaction design.It’s where Clippy was born.

Still. Talking about digital avatars.Introducing the social part of avatars.A personality. A social presence.

It’s like we have them on our social network.

WHAT ARE THE DESIGN OPPORTUNITIES THERE?

Well let’s start with at what a social machine USED to be.

Couple of things to talk about here.First, how about those jerk kids, laughing at little Timmy.They’re like this is the most lame show and tell ever.Totally ratchet.

Until they saw how AWESOME Teddy Ruxpin was.

http://www.youtube.com/watch?v=8EshrR-xk2E

MINDS BLOWN.Teddy Ruxpin totally OWNED those kids.

I wanted one of these so hard.Teddy bears are avatars of our imagination as kids.They ARE part of our social network.Teddy Ruxpin tried to make that real by adding some electronics.

It demo’d great. But it was a little creepy.And there was a key conceptual flaw.

http://thecomedycouple.com/2011/11/17/peta-hates-our-childhood/

The thing worked by cassette tape.

He only had so many lines to say.And it was limited by how many inches of magnetic tape you fed into his back.

Ultimately a mechanical machine.A device that pretended to be a social machine but wasn’t.Replaced the child’s imagination with a canned story.No longer an avatar of the child’s imagination.

Let’s look at what might happen if you made a toy that was actually social.How would we do this now?

TOYMAILtoymail.co

http://toymail.co

So let’s unpack what’s happening here.It takes the Teddy Ruxpin idea and turns it into a personal communicator.The mailmen are avatars for a child’s family.

You use the app to send a message, and it turns your voice into the mailman’s toy voice.

...and then it sends that message through.On the other side, the kid records and sends her own message.

Here’s the point.

Software makes

hardware scale.

Teddy Ruxpin played a cassette. Most talking toys do that. A fixed set of content.

But Toymail scales.Endless variety of content because it’s powered by people.

Slow:The software lets hardware be an avatar for people.It a HUGE reason we’re so attached to our phones.They’re portals to all the people we care about.

Software enables the machine to become a social machine.So it scales in that way.

But it also scales the lifespan and durability of the device itself.For Toymail, the dialogue becomes more sophisticated as the kid does.She’ll never outgrow its content.She won’t throw it out.

So connecting objects meansadding valueadding longevityadding environmental sustainability.

Same idea that motivates Apple’s Carplay or Android Auto.Where your phone drives the hardware in your car.Every time your phone gets an upgrade, your car gets an upgrade.

Outboard smarts make dumb objects both smarter and more durable.

In this case, Toymail is a portal to people.

But what about all the THINGS OR PLACES we care about?

How do we design interactionswith the digital avatars of, say, appliances?

Here’s one idea.It’s a platform called Home Chat from appliance maker LG.[start]

You text your kitchen or your appliances,have little conversations with them to make them go.

[OR EGGS! HOW MANY EGGS!]

Chat is a natural familiar way we talk to one another,so it’s easy transition to taking it to devices.And because it’s a limited command set, it’s a pretty limited vocabulary too.So the computing smarts this requires aren’t huge.

Of course if we’re talking to devices, why not just talk to them.Siri and Google Now and Cortana have started that path...

http://www.latimes.com/business/technology/la-fi-tn-ces-2014-samsung-lg-lead-smarthome-push-with-connected-appliances-20140106,0,4381746.story#ixzz2pvJV5s6g

http://www.youtube.com/watch?v=PplAymrv0hA

http://www.theverge.com/2014/1/7/5285316/lg-homechat-instant-message-control-smart-appliances-through-line

LG is now extending that idea to smart appliances with a new service called HomeChat, but it isn't focused on using your voice — it's instead based around the instant message.

The service lets owners of a new LG smart washer, drier, fridge, air conditioner, or vacuum cleaner control functions of their appliances just by sending a message. You could tell the washer to start a load of laundry, have your vacuum stop cleaning before you get home, or turn off your air-conditioner if you plan on staying out of the house.

But rather than making HomeChat a standalone app, LG is trying to build it into popular messaging services to make talking to it feel all the more natural. Right now, it's only built into Line, but LG says that it's working toward launching the service on WhatsApp and the popular Korean messaging service Kakao Talk as well.

Though the conceit is a bit convoluted — LG does have standalone apps that can perform these functions already, after all — interacting with the appliances over Line is surprisingly fun. When you start a load of laundry, HomeChat will message you back with a big sticker of a smiling cartoon washer and a bunny laying out a shirt. When the load finishes, it'll alert you with another big sticker to tell you that it's done.

...and now Amazon is getting into the game.Last month (Oct 2014) announced Amazon Echo this little wired speaker that you can talk to.

Let’s look at a warm family interaction with Amazon Echo.

[play]

Yeah, let’s just stop here for a moment to ponder this family dynamic.

Jerk dad aside, the IDEA behind it is to make it as easy to get info as possible...

Wink: Robot Butlerbitly.com/wink-butler

We all like smart technology. But when it gets TOO smart, or when it invades the most intimate spaces of our lives—even our bedroom—it gets to be too much.

Why is that?

why so

PHOBIC?ROBO

Why, Samuel L. Jackson? Why?

Machines making simplistic, poorly considered decisions.This is the foundation of every sci-fi dystopia.A fundamental fear of giving up agency and control.

Too many interfaces. If we find our phones to be so distracting, what happens if EVERYTHING is an interface? No point of quiet or retreat. We have to be gentler in the way we as designers connect with people.

Our interfaces and interactions have to take these fears and anxieties into account. As well as the simple desire to be still, unmolested.

Bend technologyto our lives.

Not the

Natural interactions with the objects, people, places in our environment.On our terms, not on the terms of screens.

When you go there, all kinds of new opportunities for interactionbetween and among our gadgets...and between people...and places...and objects.

Honor intention.

Don’t assume it.

I see this all the time in mobile.We’re great at determining the physical aspects of context.But it’s dangerous to confuse context with intent.

We need to make interfaces that respect and understandthat we don’t always know the intent,that we don’t constantly try to divine intent,but simply ask.

So it’s not just about making devices smart,so that they act on their own.

It should also be the ability to extend my will.That’s the Harry Potter stuff.Technology lets me extend my will beyond traditional physical bounds.

To express INTENTION.Let’s talk about that.

This is a theme we’ve heard throughout the day.

We tend to think that our devices are always going to keep gettingsmarter. More sophisticated.

Dumbdevices

But in fact, devices are just as likely to get dumber.And that’s a good thing.

Dumb devices doing work for us,talking to the handful of smart devices we actually interact with.

The interfaces of these dumb devices,as we’ve seen, can be very simple, possibly without screens.

[twitter]Future isn’t just smart devices. It’s a network of dumb, passive devices, too: Nest, Fuelband, Lumo etc.[/twitter]

Visa and MasterCard both have credit cards with screens and keypads.Visa unveiled theirs in Europe in 2010.MasterCard unveiled theirs in Singapore in November 2012.

Idea is to make online payments more secure by ensuring you’re holding the actual card, not stolen card numbers.

You make up a numeric password and type it into the card.And then you type that same number into a web interface.

Speaking of little displays...

http://www.visaeurope.com/en/newsroom/news/articles/2010/visa_codesure_gets_green_light.aspxhttp://newsroom.mastercard.com/press-releases/mastercard-introduces-next-generation-display-card-technology-a-first-for-singapore/

The rumor mill is going full-tilt that Apple will release an iOS watch this year.

How does your website or app look on THIS?

And even this seems easy in comparison to other stuff that’s coming at us.We’ve gotten pretty good over the past few years withfitting our content and messaging into different squares and rectangles.

Phones, tablets, etc.

But then...

Whoa! Google Glass. What the hell?

How do we deal with that.How do we allow people to experience our sites and appson such incredibly different devices?

The pebble watch, google glass,THESE ARE SIMPLY ALTERNATE WINDOWS INTO OUR PHONE.Convenient, context-specific displays of apps and data available on the phone.Glass plugs into your Android phone. It’s the phone driving the logic.So we’re back to that Corning vision of the world.

A smart device pushing a UI to relatively dumb displays.

Right now, we’re badly equipped for this. Our content is a mess,not ready for the multidevice present, let alone the future.

How do we design content and experiences for devices we haven’t even imagined yet?If we can’t know the future, we can’t be future-proof,but we can at least be future-friendly.

And frankly, that means starting with our content.

Your APIis the

application

It’s the underlying service that we need to provide,focus on the format of the raw content.

Chunked up, well described, stripped of presentation,so that even dumb devices can take the content they need and present it appropriately.

If we can do a better job of structuring our content,describing it,so that we can have meaningful APIs that let devices grab just the content they want,

that’s creative control in an uncertain world.Because all of what I’m talking about hereis juggling a growing number of gadgets, smart and dumb,in our personal lives.

A cloud of social

devices

So our final stop on this journey lands us here.In the cloud.

The cloud as we know it is part of it.A lot of this stuff has to be stored and accessible out in the interwebs.

But the cloud I’m talking about is a cloud of social devices.You take all these trends:sensorsmirroringremote controlmigrating controlpassive interfaces

and you have a set of truly social devices that are in our face when we need them,out of the way when we don’t but still doing our work.

So that means that we are doing a lot more than building applications.Those are just the container.We’re building services.

[twitter]Future UX: design for sensors, design UI that migrates among devices and off devices completely. No screen required.[/twitter]

You have to think beyond the single interface.That’s what occupies us, though.

Look beyond the one application you’re working on today.That’s just a single window into your service.

Applications are just containers.Temporary containers: because presentation deprecates.We throw these containers out all the time.Websites, apps of two years ago, out of date. Need new presentation.

Service and content is more lasting.Think about services... Design services.

Whole systems that can support lots of devices.Because that’s the way to prepare for this future of social devices.

What I’ve tried to do here.Pull together the themes that we see now.

[next]This is imperfect. Making these connections is hard.Steve Jobs: “You can’t connect the dots looking forward, you can only connect them looking backwards.”

Still: This stuff already in labs.And I think we can see enough of the coming changes that we cansee what we have to start building nowto be ready in three or five years.

[twitter]Steve Jobs: “You can’t connect the dots looking forward, you can only connect them looking backwards.”[/twitter]

★Push sensors

★think social

★Your ecosystem

★ We’re all cloud developers

★ Your api is the application

★New input methods

★The future is here

This is where we are now, right? This is where we began.So how do we prepare for where we’re going?

Sensors are the superpowers.Don’t do it just because. Don’t force maps into your app just because of GPS.But be thoughtful. What does this let me do with my data.Or to save time.

Think social. For once, not FB or Twitter.Interact with other devices. What are the opportunities to share not only content but control.Start thinking about that future of social devices.

Your ecosystem.Most of us are not Apple or Amazon or Microsoft or Google.We don’t get to control the big-picture ecosystem.But we can control our own. You are not building an app.You are building a system. Prepare for multidevice future.

This means we have to think cloud. Most of us are. But it’s crucial.Web apps vs native apps a constant debate.Important decision and conversation for builders.But less important for our users. It’s an implementation detail for them.Web, native, browser, app, iPhone or Android,it all converges in the cloud. The data and interaction is there.

New inputs. Start planning for the future of speech, gesture, facial recogntion.Is your content ready for this? How do those things work in combination?

The future is here.Bill Buxton’s 20 year rule.

But we’re shaping that future together. Right here in this room.Exciting time. Best jobs. Make something amazing.

Maythe force

be with youJosh Clark

@globalmoxiebit.ly/beyond-mobile

[twitter]Slides for my talk “Beyond Mobile: Where No Geek Has Gone Before” (pdf): http://j.mp/beyond-mobile[/twitter]

Designing for Touch... and Beyond - Josh Clark

Design

Transcript of Designing for Touch... and Beyond - Josh Clark