Direct manipulation is broken

Why the IoT asks consumers to think like programmers, and the UX challenges this creates

Claire Rowland / @clurrO’Reilly Design Conference21st January 2016

“This is more than a UX book; it covers all of the critical design and technology issues around making great connected products.”

DAVID ROSE, ENCHANTED OBJECTS

“As a grizzled veteran of several campaigns within the matter-battle of the Internet of Things, I was pleasantly surprised to find the number of times this book made me pause, think, and rethink my own work (and that of others). A very valuable addition to the canon of design thinking in this emerging area.”

MATT JONES, GOOGLE

“Connected products are a unique beast. Designing this type of product well takes lots of iterations & user testing. This book takes you through the paces. We keep one on the shelf at Highway1 for

our IoT companies.” BRADY FORREST, HIGHWAY1.IO

“The IoT breaks direct manipulation. Using connected devices is conceptually more like programming.”

Alan Blackwell Professor of Interdisciplinary Design, Computer Laboratory, University of CambridgeVery roughly paraphrased

“The IoT breaks direct manipulation.

Using connected devices is conceptually more like programming.”

Direct manipulation

“A human–computer interaction style which involves continuous representation of objects of interest and rapid, reversible, and incremental actions and feedback.”Ben Shneiderman “The future of interactive systems and the emergence of direct manipulation". Behaviour & Information Technology

Why direct manipulation works

- Visibility. By looking, the user can tell the state of the device/system and the alternatives for action.

- A good conceptual model. The designer provides a good conceptual model for the user, with consistency in the presentation of operations and results and a coherent, consistent system image.

- Good mappings. It is possible to determine the relationships between actions and results, between controls and their effects, and between the system state and what is visible.

- Feedback. The user receives full and continuous feedback about the results of actions.Chapter 2, The Design of Everyday Things

A key condition of direct manipulation is that the user’s actions and the results happen at the same time, in the same situation

Remote control

Image: British Gas

Automation

Scheduling things to happen in future, like turning on lights to welcome you home, or turning off unnecessary appliances to save energy overnight.

Image:; Martin Cathrae via Flickr

Interoperable, coordinating productsVideo: shot at Target Open House

Things in your world will work together to meet and maybe even anticipate your needs.

A sleepsuit that detects your baby waking up could trigger a morning routine.

An “intelligent” security system could draw on information from your calendar, to detect possible intrusions.

All these scenarios break direct manipulation because they displace user actions from system responses

Remote control: displacement in space

You might:

- Set the security alarm when there’s still someone at home

- Turn on lights when they’re trying to sleep- Turn the heating off when someone needs it

- …or turn on an electrical appliance when there’s a gas leak in the home

Image: Audetat via WikiCommons

Risks of displacement in space

- The action may be inappropriate or unsafe at that time (or just annoying to someone else)

- Users may not see the consequences of their actions

- Users may not realise that the action they thought they took didn’t work as planned

Images: https://tiare75.wordpress.com

Is that 7pm where I am now, or at home?

Automation: displacement in time

- That “unnecessary” appliance that gets turned off might be the tumble dryer running overnight to ensure a shirt is ready for an important meeting.

- Your alarm goes off because your calendar says you’re not ‘supposed’ to be home and infers you might be an intruder.

- Your baby wakes up super early and you don’t want to start the day, you want to GO BACK TO SLEEP

Google Image search for baby waking early

Risks of displacement in time

- Users need to anticipate their own future needs: which humans are not very good at.

- Users may forget that they have set up an action, and then be negatively affected by the execution of that action.

- They need to anticipate all the changing conditions to which the system might have to respond.

http://www.independent.co.uk/sport/rugby/rugby-union/international/barbarians-vs-samoa-interrupted-by-sprinklers-as-fans-criticise-no-beer-lack-of-wi-fi-and-poor-10478187.html

Flexible, multi-purpose hardware

Automated security lighting schedule?

Monitoring energy consumption of an appliance?

Remote control of your kid’s Xbox?

Image: Belkin

Functionality can be modified in a remote software layer

Smart plug that ran an automated schedule for a lamp is moved to hair straighteners…

and is now turning those on and off when you’re not at home

Image:; London Fire Brigade

Flexible, multi-purpose hardware: displacement in function

Risk of displacement of function

- The behaviours of the device that were appropriate for its original use are ported over to a new use, and are now actively dangerous

Image: Shropshire Insurance

But hey, that’s why we need to make things smarter, isn’t it?

- You could have a gas sensor that alerts you and turns off the electricity if it detects a leak.

- You could come up with a way of temporarily excluding the tumble dryer from your energy saving program.

- You could program the smart plug to identify changes in energy consumption that might indicate that it’s being used with a different device, and disable the schedule.

- You could disable the alarm with two factor voice authentication: not just keycode or phone, but voice.

- You could have that baby’s room macro run only after 6am.

Image:; meetangee.com

Indeed you could.

But it’s very hard to mitigate every possible problem.

Number of edge cases you’d have to anticipate

Few

Tons

Discrete product Many interconnecting products

“…the cumulative complexity of a bunch of simple things—regardless of how delightful, simple and desirable they may be—will soon exceed the ability of humans to cope.”

Bill Buxton Principal researcher, Microsoft Researchhttp://www.fastcodesign.com/3054433/design-moves/the-most-important-design-jobs-of-the-future

In Norman’s terms, the design upshot can be:

- Poor visibility. It’s not always possible to tell, by looking, the state of the device or the actions that are possible with it.

- A complex conceptual model. The conceptual model of a connected product is way more complex. Need to understand parts, what each does and how they interrelate. Separation of operations and results can make it harder to identify consistencies.

- Obscured mappings. It is harder to determine the relationships between actions and results, between controls and their effects, and between the system state and what is visible.

- Poor feedback. The user may get only partial, delayed, incorrect or intermittent feedback about the results of actions.

“The IoT breaks direct manipulation.

Using connected devices is conceptually more like programming.”

Since when did turning lights on and off count as programming?

Images Keil C51 C Compiler (Phaedrus Systems), Philips

Let’s break down a home automation task…

I want a nice green lawn Understand and frame your own problem.

= Requirements definition

Image: Adam Kerfoot Robverts via Flickr

“I need a controller, and some sprinklers, and I need to plan around weather, water use restrictions, and social events”.

= Specification

Image: Blossom

“Now… how do I get this thing to do what I want?”

Image: Rachio Iro

= Implementation

“Oops. What went wrong?”

=Testing and debuggingImage: http://blogs.ifas.ufl.edu/pinellasco

This is software engineering!

Users must:

•Anticipate their own future needs and plan a solution•Configure the system to meet these needs using an abstract representation

•Deal with unforeseen conditions that may make the system behave in unexpected ways

•All the while… considering future conditions that might cause unexpected effects or conflicts

Requirements definition

Specification

Implementation

Testing/debugging

You say that like it’s dirty.

Empowering people to program is great

But it’s a lot of work for a consumer product

Product Tool

In areas where they don’t have expert knowledge or are short on time, consumers tend to look for products which come preconfigured to solve a specific problem, rather than multi-purpose tools which require the user to define and solve their own problem.

It’s a time and attention issue

Image: XKCD

OK.

But that still doesn’t explain how using a simple remote control is like programming

Users need to engage with the system model

InternetLocal WiFi

Programming can be hard, and painful, however good at it you are

Experts *expect* to make mistakes (and are often paid to do so)

I expected it wouldn’t be perfect the first time around and there is always an exception. You learn about them after you start using them.

Expert home automation user (and IT security professional)

Image: Zach Luz (zachluz.com)

…users don’t (and are never paid to do so)

- Things not working isn’t part of the normal path to success for a consumer product… but this will happen

- It’s especially painful at home: which is supposed to be our haven from the world

- System designers need to do all they can to mitigate potential damage

The logic of even apparently quite simple things can be complex… and inconsistent

if (time == 1800) then set light =on if (time == 2200) then set light =off

while (mode ==scheduled) then

{ while (time >=1800 && time <=2159) then set target temperature =72F while (time >=2200 && time >=0659) then set target temperature =65F while (time >=0700 && time >=0859) then set target temperature =72F while (time >=0900 && time >=1759) then set target temperature =58F }

What do we do about it? Or, how can we make controlling complex systems easier?

Home Maestrohttp://research.microsoft.com/apps/pubs/default.aspx?id=70594

This is not (just) about new types of programming interface

They might help with the implementation part of the problem.

But they don’t help with the problems of predicting your needs, planning and debugging, forming a conceptual model or knowing what’s happening at a remote location.

Images: Lowes

Better feedback: was my command executed or not?

Better visibility/feedback: remote context

- Am I about to do something that might be unwise?

- Has something happened that might not be what I want?

Carrier 12:00 PM

!! Jen is at home !!

ARM SYSTEM

Home security alarm

Better visibility: why did that thing happen/not happen?

- “Why didn’t the dishwasher run this morning?” (perhaps the smart electricity grid turned off the supply to that nonessential device at a time of peak demand)

- “Why is the hall light flashing on and off?” (perhaps it is receiving conflicting instructions from security and energy saving apps)

http://www.cs.cmu.edu/~NatProg/whyline.html

Error prevention

- Flag up potential consequences: sandbox simulations to check for rule conflicts

- Try to prevent the user from doing dangerous things

Your new lighting rule conflicts with your

security rule

I’m pretty sure that’s no longer a lamp I’m

powering

What about autonomous systems?

Scott Jenson: Home automation is an easyhard problemhttps://jenson.org/easyhard/

…common sense is hard to get right

What about us?

Will we adapt to a programming-like world?

[Ben Evans]

My grandfather could probably have told you how many electric motors he owned. There was one in the car, one in the fridge, one in his drill and so on.

My father, when I was a child, might have struggled to list all the motors he owned (how many, exactly, are in a car?) but could have told you how many devices were in the house that had a chip in.

Today, I have no idea how many devices I own with a chip, but I could tell you how many have a network connection. And I doubt my children will know that, in their turn.

Benedict Evans Andreessen Horowitzhttp://ben-evans.com/benedictevans/2014/5/26/the-internet-of-things

If you’ve enjoyed this, the book covers:

- How the technology of IoT affects UX- Product and design strategy for connected devices- Industrial design- Interface and interaction design for embedded devices- Cross-device interactions and interusability- Interoperability- Responsible IoT design- Designing with data- Prototyping and user research methods for connected products

www.designingconnectedproducts.com@clurr | claire@clairerowland.com