How People Actually Use Thermostats

Quantifying Usability in Programmable Thermostats

Alan Meier* and Cecilia Aragon – LBNL

Therese Peffer and Daniel Perry – UC Berkeley

Marco Pritoni – UC Davis

* akmeier@lbl.gov

Thermostats

• In 2008 Energy Star concluded that homes with programmable thermostats were using more energy than homes with manual thermostats.

• Energy Star terminated the program and began developing a new specification

Premise:   Improving the usability of thermostats will facilitate energy‐saving behavior

This Talk

1. Surveys of usability of thermostats in homes2. Methodology for quantifying usability of 

thermostats and other controls

Conclusions:• Few homes exploit full potential of programmable thermostats 

• It is possible to quantify usability in a way suitable for use in design specifications for standards

Are Programmable Thermostats Used Correctly (or features fully exploited)?

Parallel investigations via:

1. Weatherization crews

2. Amazon Mechanical Turk

3. Interviews and other on‐line surveys

-“Do you use the programming features of the thermostat?”

- “Yes”

-“Are you satisfied with your thermostat?”

- “It’s OK”

-“If you could have a new thermostat, what would you like it to do differently from your current thermostat? (language, font size, button size, technical terms)?”

- “NO”

“The thermostat is on Hold”

Survey of a Weatherized Home

Weatherization Survey Results

• 20 homes visited (in mid‐west)

• 45% on “hold”• 5% switched off (in winter)

Talk to us if your wxncrew would like to participate in Round 2 of  this survey

Amazon Mechanical Turk Survey

The Workers:

1. Fill out surveys on the web

1. Photograph their thermostats

1. Get paid via Amazon

We get rapid, tabulated, results!

Amazon Mechanical Turk Results

• 63 responses in 24 hr

• ~20% had errors in time setting

• ~50% on long‐term hold

• Next survey will collect several hundred responses

Examples of uploaded photos

And Now the Video…

How to Measure a Product’s Usability?

1. Define tasks

2. Quantify peoples’ ability to accomplish tasks

3. Compute “score” based on metrics

4. Compare to reference model

Details of Usability Tests

• 5 thermostat interfaces• 31 participants• 2 interfaces per person• 6 tasks for each test• 372 videos

Thermostats Tested

Define Tasks

Task 1: Turn the thermostat from “off” to “heat.”Task 2: Set the correct time.

Task 3: Identify the temperature the device is set to reach.

Task  4: Identify what temperature the thermostat is set to reach for Thursday at 9:00 PM.

Task 5: Put the thermostat in “hold” or “vacation” to keep the same temperature while gone.

Task 6: Program a schedule and temperature preferences for Monday through Friday.

Task 1: Turn the thermostat from “off”to “heat.”

Distribution of Times for Subjects to Complete Task 1

Task 1: Turn the thermostat from “off” to “heat.”

Average Time to Complete Task by Model

Task 1: Turn the thermostat from “off” to “heat.”Task 2: Set the correct time.Task 3: Identify the temperature the device is set to reach.

Time to Complete Tasks by Model

Consumer Reports Recommended the Model that Ranked Lowest in Usability

Conclusions

Current usage of thermostats:– ~50% of programmable thermostats set to long‐term hold

– ~20% do not have the correct time

Many homes fail to use features of programmable thermostats

– Do people believe that installing a PT will automaticallysave energy?

We created and demonstrated a methodology to quantify usability

– Applies to HP water heaters, TVs, lighting controls, etc.

Premise:   Improving the usability of thermostats will facilitate energy‐saving behavior

Future Work

• Expand surveys

• “Road Test” methodology

• Next steps in measuring usability3. Compute “score” based on metrics4. Compare to reference model

• Apply methodology to lighting, TVs, other appliances

Talk to us if your wxncrew would like to participate in Round 2 of  this survey

Alan Meier akmeier@lbl.gov

Back‐Up Slides

akmeier@lbl.gov

Controls for Heat Pump Water Heaters (HPWHs)

1tute, Inc. All rights reserved.

Controls

GERheem

A. O. Smith

An incorrect setting could increase electricity use 1500 kWh/yr

Codes for Hunter Thermostat

HTR Codeen = energy button tu/td = set temperature up OR down

ht = home today button h/r = hold/ return button

tu = temperature up p = program button

td = temperature down pd = program day button

ct = current temp display on touchscreen [0] blank touchscreen space

ti = display time on touchscreen f = filter button

sun = sun icon on touchscreen fo = fan on switch

he = heat switch fa = fan auto switch

d/t = day/time button fof = fan off

“Lostness”: A Metric for Quantifying Usability

Minimum Keystrokes to 

Accomplish Task

Actual Keystrokes by 

Subject

cl C

cu/cd sy

c d

sr

c

cl

Cd x  10

cu

Cd x 3

Cu x 8 

.

.

.

(3 keystrokes) (28 keystrokes)

“lostness coefficient”= 28/3~ 9

Keystrokes transcribed from videos

Why Do People Keep Thermostats on Long‐Term Hold?

• They don’t– The survey captured a rare event

• They don’t know how to easily operate other modes (including leaving this mode)– Usability problems

• They think this mode is energy‐saving (since they installed an “energy‐saving” thermostat)

• They want this setting– Technical implications about optimal design of controls, levels of efficiency investment

Mechanical Turk – ScreenshotsSurvey General

Settings

Mechanical Turk – ScreenshotsResults: Web View