Measuring wireless network usage with the Experience Sampling Method Tristan Henderson, Denise...

Measuring wireless network usage with the Experience

Sampling Method

Measuring wireless network usage with the Experience

Sampling Method

Tristan Henderson, Denise Anthony, Tristan Henderson, Denise Anthony, David KotzDavid KotzDartmouth CollegeDartmouth College

Tristan Henderson, Denise Anthony, Tristan Henderson, Denise Anthony, David KotzDavid KotzDartmouth CollegeDartmouth College

WinMee, April 3 2005Measuring wireless network usage with ESM

Motivation• Measuring wireless networks is useful

• deployment, protocol development, application development, mobility modeling, etc...

• So lots of people have done it• Network-side measurements of lots of WLANs: IBM,

UNC, SIGCOMM, Georgia Tech, Stanford, UCSD, Dartmouth, ...

• Network-side measurement only gets you so far• Tells you what happens, but not why it happens

• To know why, need to ask wireless users themselves• But reliable polling can be difficult


The Experience Sampling Method

• A psychological ‘diary method’• subjects self-report their own experiences

• potentially-fewer biases since no interviewer involved

• signal subjects at random times and ask them to report• timely reports means users don’t forget their experiences

• interval-contingent: subjects report at regular intervals• signal-contingent: subjects report when an event occurs

• ESM reports: subjects fill out questionnaires• questionnaires ask about current activities, conditions,

feelings• finite time period, e.g., seven signals per day over

seven days


Experimental setup - users• Dartmouth WLAN

• ≥560 Cisco/Aruba 802.11a/b/g APs• complete campus coverage• ongoing monitoring since 2001 (syslog, SNMP,

tcpdump, CDR)

• 30 subjects• all undergraduates, 15 male, 15 female• all owned wireless laptops

• 23 Windows, 7 Macs

• recruited via website• offered $100 each

• subjects provided ‘conflict times’: times not to be signaled


Experiment setup - technical• 30 participants, 30 alphanumeric pagers, 30

notebooks• Motorola Bravo 406-512MHz pagers

• considered PDAs or 802.11 devices• but wouldn’t work off-campus or when network down

• Notebook: signal- and event-contingent questionnaires• different questionnaires depending on type of alert• questions incl: devices/applications in use, location, current task

• 7 day study, 7 signal-, ≤3 event-contingent alerts/day• signal-contingent alerts at random times (excluding conflicts)

• minimum 45-minute interval between alerts

• monitored subjects’ wireless MAC addresses and sent event-contingent alerts if certain events occurred• high throughput, high errors, ping-ponging, association with busy AP


Wireless vs overall mobilityUsers were paged at up to 6 locations in a day, but were only on WLAN at ≤ 2

Do we need better coverage?


Usage locationsWireless used more in homes, libraries

% o

f ale

rts

at

loca

tion


Applications in useUnsurprisingly, P2P was big bandwidth hog - our sample is representative of whole campus


Wireless vs wired behaviour• Wireless behaviour is multi-modal

• when using wireless laptop, more than one device in use

• More likely to be using cellphone and/or TV when on wireless

• Fewer high-bandwidth activities (filesharing, streaming media) on wireless than on wired

[Wireless] tends to be a little bit slower than wired... I download a lot of music so that can be frustrating.““ ””


Home vs non-home behaviour

• Both home and non-home:• e-mail• work-related web-browsing

• When at home:• P2P, IM• non-work-related web-browsing

[Wireless] lets me move my study space to wherever I want it to be.““ ””


Communication patterns• Locality

• 83% of email/IM is local• 86% of phone (analogue/cell/VoIP) is non-local

• VoIP• 65% of subjects completely unaware of VoIP• VoIP users did not use cell phones, and vice versa• (but we have very little cellphone usage in

general)

If the quality and price of VoIP phones were the same as the cell phone, I think I would definitely use them.““ ””


Lessons learned• Dealing with subjects is hard work!

• choose time carefully• avoid student examinations, vacations• allow time to collect info (MAC addresses etc.)

• expect schedules to change• students change conflict times with 5-minutes notice...• one laptop broke and was replaced during the week

• explain parameters clearly• one user went home (to a different state) but thought she

could still be monitored on home WLAN

• automatic participant pre-screening• we used face-to-face interviews• could monitor wireless activity prior to participant-

selection


More lessons learned• Consider all event-contingent alerts

• we thought it would be less intrusive to limit to 3/day• ended up with very few event-contingent alerts

• ignore 45-minute alert interval• interval meant we ignored some interesting wireless

events

• Use a two-way signalling channel• lots of pages failed, but we didn’t know• no real-time feedback

• users would complain, but only at the end of the day (when they hadn’t received sufficient pages)

• two-way pagers?• 802.11 devices inappropriate


Related work• Lots of ESM studies in psychological literature

• teacher job satisfaction• youth happiness• experiences of new parents

• Communications-based ESM studies• Ubicomp - Intel Personal Server (Consolvo & Walker

2003)• Virtual Environments (Gaggioli et al. 2003)• Television-viewing habits (Kubey et al.

1986/1987/1990)

• Most studies are signal- or event-contingent• Our study attempts to combine both


Summary• As well as measuring wireless networks,

need to measure wireless network users• ESM provides a technique for measuring

user experiences• allows us to see when users aren’t on the WLAN• allows us to see more detail about what users are

doing• allows us to see usage of non-802.11 devices

• ESM studies are difficult to set up• difficult to deal with users

• Future work: larger study that benefits from the lessons learned in this study

Measuring wireless network usage with the Experience Sampling Method Tristan Henderson, Denise...

Documents

Transcript of Measuring wireless network usage with the Experience Sampling Method Tristan Henderson, Denise...