WHAT PROBLEM DOES LIVELABS SOLVE?
Transcript of WHAT PROBLEM DOES LIVELABS SOLVE?
25/7/2014
1
LiveLabs: Building An In-Situ Real-Time Mobile Experimentation Testbed
Talk at HotMobile 2014Santa Barbara, Feb 27th 2014
WHAT PROBLEM DOES LIVELABS SOLVE?
Amazing mobile sensing app!!MobiSys deadline < 3 weeks!How to test with real users?
Lets just test it with lab usersClaim it is “real-world”Reviewers won’t know better
Wow! They did know better!Need access to real venuesWith real users on real devices
HOW???
25/7/2014
2
LIVELABS IN ACTION (WHEN OPERATIONAL)
30,000 opt-in consumers
Retail & Consumption
Leisure & Tourism
Telco & IDM
Multiple Urban Venues & Lifestyle Verticals
Mall@Singapore SentosaChangi AirportSMU
Resource-efficient deep context
collection
Real-time mobile analytics & insights
Real World experimentation
LIVELABS: PARTICIPANTS & VENUES
25/7/2014
3
LIVELABS DATA FLOW
Internet Cloud
Real-timeAnalytics Server
ExperimentationServer
Results Server
Investigators
LiveLabs Urban Lifestyle Innovation Platform
LiveLabs ContextCollection application Installed in smart phones
External Analytics Providers(eg. LARC,
IBM, Accenture,..)Specify
Interventions
SPECIFYING EXPERIMENTS : CURRENT THOUGHTS
What location (e.g., SMU, 1st
floor of Mall) does the participant need to be on
STATIC Demographic criteria (age, gender, occupation, etc.)
DYNAMIC Context Criteria (e.g.., person sitting down in the foodcourt)
Intervention—e.g., a pop-up “Notification” on screen with a discount coupon
A Simple Web-based Interface forSpecifying Targeted, Context-Based Lifestyle Experiments/Services
25/7/2014
4
CURRENT STATUS
• LiveLabs@SMU operational since Sep 2012 (actually Apr 2012) • Approx. 2000 participants signed up; approx. 700 active participants• Data collection for Android and iOS platforms deployed
• Campus-wide Indoor Location Tracking• Longitudinal traces of over 4000+ individual devices using server-side location• Controlled activation of fine-grained client-side location (Android)
• Developed Analytics over Mobile Data• Queuing Detection: Research prototype tested• Group Detection: Under active R&D
• Interventions/Promotions• Merchant promotions provided to participants via in-house built SMUddy App• First end-to-end experiment to run next week!
• Fair Amount of Visibility These Days ( Double edged)• Appeared in slides by Samsung US Researcher yesterday morning (woo hoo!!)
1. Deep, energy-efficient, continuous, context collection
2. Continuous indoor location tracking in public spaces
3. Derive Deep Analytics from Context
4. Run automated social experiments on mobile devices
5. Handle transient network traffic loads
LiveLabs: Key Component Technologies
• Clients for Android, iOS, Phone8 .• Server-controlled capture of phone events (e.g., SMS, URLs) & sensor data
• Client-side +-3m accuracy for Android.
iOS
• Client-side +-3m accuracy for Android.• Server-side tracking for all platforms (e.g., iOS, Phone 8) ~ inter-AP distance
• Real-time Queue Detection System.• Detection of Dynamic Groups from Spatiotemporal trajectories
• Intervention Management Portal (v1)
ads/promotions.
• Intervention Management Portal (v1) allows location & time-based delivery of ads/promotions.
• Use of TV Whitespace and real-time RF Mapping technologies under investigation
Key Research Challenges Current Innovations/Capabilities
25/7/2014
5
RESEARCH HIGHLIGHT 1: QUEUE DETECTION
• New urban applications
• “Find the vendor withshortest waiting time”
• “Provide discounts tocustomers with long expected wait times”
• “Trivial” with fine-grained location info
• Not possible in practice
• Solution: Use accelerometer and infer queuing
• Does it actually work?
Measured Service Times are Highly Variable
QUEUE DETECTION PERFORMANCE
Still Possible to Achieve Good Results
F&B Outlet on Campus F&B Outlet on Campus
Tested on 15 different occassions at various locations (coffee shops, movie theaters, taxi stands, etc.) in Singapore and Japan
25/7/2014
6
RESEARCH HIGHLIGHT 2: GROUP DETECTION
• Fast and accurate group detection is very useful!• For timely recommendations and contextual reasoning
GROUP DETECTION PERFORMANCE
• Existing Solutions are not very effective• Trajectory analysis? Limited location infra + collocation of non-group
people
• Bluetooth scans? High power use + miss-detection (esp. when crowded)
• Semantic transition and sensor-driven features are effective!
VenueRecall(%)
Precision(%)
CoEX Mall 61.16 92.15
PlazaSing 68.48 97.39
* at 10 minutes latency
25/7/2014
7
ALL THAT GLITTERS IS NOT GOLD
• Taken a long *long* time to get to this stage• Idea was conceived in late 2009
• Funded in 2011
• Launched in 2012
• First “complete” test with actual users Next Week
• Quite a few technical and administrative challenges
• Most are “Obvious” in hindsight
• Let me share our pain so that you won’t have to go through it
CHALLENGE 1: INDOOR LOCATION• Our naïve initial position
20 years of work in this area. Must be solved! Lets just reuse what others have done
• Realisation: How come the results are not great? Heterogeneity of devices and environments makes the accuracy much *MUCH* worse!
RSSI is a terrible input Closed platforms make sensor fusion quite hard
• Big takeaway (wisdom so far) Inter-AP distance accuracy possible with low energy
across all devices (server-side techniques) Better accuracy very hard without significant energy and
programming costs (sensor fusion etc.)
25/7/2014
8
CLIENT SIDE ANOMALIES (AP INCONSISTENCIES)
Procedure: Use Same Device (Galaxy S 4) at the Same Place at the Same Time Measuring the Same Set of APs
Implication: Fingerprints / Models Need to be quite Dynamic!
AP 1 : Day 1 Highest AP 2 : Day 2 Highest AP 3 : Day 3 Highest
CLIENT SIDE ANOMALIES (PROXIMITY EFFECTS!)
Procedure: Measure each device individually at the same spot for 2 mins. Then measure both devices side by side for 2 mins.
Implication: Not Sure! Makes location much harder for groups!
Expected Result S4s Have Switched Gains!
S3s and S4s Have The Same Effect
25/7/2014
9
SERVER SIDE IS NOT MUCH BETTER!! (SIGH..)
Procedure: Measure RSSI at AP over a period of 3 hours. Device is connected and left at main screen (allowed to lock / power down)
Implication: Some devices are very hard to track! (lack of updates)
Some devices are quite noisy (errors in location)
Note 2: Lots of Updates + Stable
HTC One: Lots of Updates but NOISY!!
iPhone 5S: Few Updates & Noisy!
CHALLENGE 2: CONTINUOUS SENSING• Our naïve initial position
Lets turn everything on at full rate and collect the kitchen sink!
How hard can it be??
• Answer: Very Hard!!
Energy cost of individual sensors is large
Energy cost of multiple sensors may not be linear
Energy cost of multiple tasks is dominated by the most expensive task
Heterogeneity of devices and tasks makes situation much *much* harder!
25/7/2014
10
ENERGY COST OF INDIVIDUAL SENSORS
0
20
40
60
80
100
120
140
160
slowest slow fast fastest
Po
we
r C
on
sum
pti
on
(m
W)
Sensing Rate (4 default modes on android)
AccelerometerGyroscopeCompass
IT GETS WORSE WITH PROCESSING & STORAGE!!
0
50
100
150
200
250
300
350
400
450
500
slowest slow fast fastest
Po
we
r C
on
sum
pti
on
(m
W)
Sensing Rate (4 default modes on android)
Accel with Internal Flash StorageAccel w/o Internal Flash StorageLight with Internal Flash StorageLight w/o Internal Flash Storage
2x higher
5x higher!!
25/7/2014
11
ENERGY COSTS MAY NOT BE LINEAR
0
200
400
600
800
1000
1200
1400
slowest slow fast fastest
Po
we
r C
on
sum
pti
on
(m
W)
Sensing Rate (4 default modes on android)
All inertial sensors (accel, gyro, compass)Inertial + location + others (pressure, light)
no differenceLarge sub linear increase
Large non linear increase
CHALLENGE 3: PARTICIPANT RETENTION• Our naïve initial position
This is so cool & We LOVE IT!! Surely, everyone will opt in and stay!
• Realisation: No they won’t!
Drop out rate was very high at the start
Even with payment incentives!! (not obvious initially)
One main reason was lack of compelling usage
Solution: Develop Cool in-house customised apps
Challenge: Cool is relative and developing apps requires a lot more work
25/7/2014
12
COOL APP 1: SMU BUDDY (SMUDDY)
Heatmaps Showing Free Spots on Campus
Friend finder + location and time based
messaging
Exclusive promotions from Campus vendors
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Au
g 1
3A
ug
13
Se
p 1
3S
ep
13
Se
p 1
3S
ep
13
Se
p 1
3O
ct 1
3O
ct 1
3O
ct 1
3O
ct 1
3N
ov
13
No
v 1
3N
ov
13
No
v 1
3D
ec
13
De
c 1
3D
ec
13
De
c 1
3Ja
n 1
4Ja
n 1
4Ja
n 1
4Ja
n 1
4Ja
n 1
4F
eb
14
Fe
b 1
4F
eb
14
Fe
b 1
4
No
. Of
Pa
rtic
ipa
nts
Month of the Year
Registered Users
Users Installing LiveLabs Applications
THE SLOW GROWTH TO “SUCCESS”Opportunity!!<small print>Or major problem <\small print>
25/7/2014
13
MOST SERIOUS CHALLENGE: ADMINISTRATION• Our (Archan and myself) naïve initial position
We can run everything (people, vendors, research) ourselves
• Realisation: No we can’t
This lab is like a startup. Way too many moving parts
Almost burnt out handling everything
Production is quite different from research Client management takes *SOOO* much time!
Solution: Hire dedicated PMs, lab manager (accountant), business development managers
LIVELABS: LESSONS LEARNED UP TO NOW (SUMMARY)
• Technical• Indoor Location Tracking is Not a Solved Problem
• Too many real-world anomalies with existing techniques
• Cannot do Continuous Mobile Sensing• Large amounts of low fidelity sensing with burst of high fidelity sensing
• The Tail Really Does Matter!• Venue operators prefer solutions with no fluctuation (even if base is
worse)
• Administration• Attracting Participants is Easy, Retention is Hard!
• Need to find what motivates participants to stay on (apps in our case)
• Production, Research, and Administration Do Not Mix!• Needed separate teams for each to ensure quality and prevent burnout
25/7/2014
14
ACKNOWLEDGEMENTS
Building something like LiveLabs requires standing on the shoulders of many other people
• Faculty – Archan Misra, Youngki Lee
• Post-docs – Rijurekha Sen, Victor Lu
• Ph.D. Student – Kartik Muralidharan, Sougata Sen, Joseph Chan, Nguyen Huynh
• Professional Staff – Jonathan Wang, Kenneth Fu, Kazae Quek, Sipei Huang
• Engineering – Swetha Gotipati, Le Gai Hai, Kasthuri Jayarajah, William Tan, Jeena Sebastian, Vignesh Subbaraju, Nguyen Minh, KohQuee Boon, Sriguru Nayak
• Many more people + interns have worked with us over the years
SUMMARY
• LiveLabs aims to change 4 real-world venues into living testbeds
• Real people with real devices in real environments performing real actions in real time
• We are live at SMU and going live at the airport soon
• 2000 sign ups at SMU
• 1st end-to-end experiment going live next week
• Free to use and open to all (at least at SMU)
25/7/2014
15
FOR MORE DETAILS & TO USE LIVELABS
Come talk to myself, Archan, or Youngki.
Or contact me at [email protected] and/or visit
http://www.livelabs.smu.edu.sg
Come test your applications, sensing technologies, and other mobile phone related solutions with us
We are hiring! Post-docs, research engineers, and Ph.D. students (in all areas of systems development and
research)
Please contact me if you are interested.