Gabriel: Towards

Wearable Cognitive Assistance

Kiryong Ha, Zhuo Chen, Wenlu Hu, Wolfgang Richter,

Padmanabhan Pillai, and Mahadev Satyanarayanan

Faces Daily Routine

Cognitive Decline

20,000,000+ Americans affected

Ron traumatic brain injury

Alzheimer’s disease

mild cognitive impairment survivors of stroke

One-month delay in nursing home

admissions saves $1,000,000,000+/year

10/24/2014 2

Text

Can Wearable Technology Help?

10/24/2014 3

Continuously capture, interpret, give guidance and

System Architecture

10/24/2014 4

“Please stop and check traffic”

“Barack is saying hello to you”

“Your dog wants to go out for a walk”

Why Today? Advances in 3 Independent Arenas

10/24/2014 5

Algorithms

Offloading Technology

Wearable Hardware

Challenges – Architecture

1. Crisp Interactive Response

2. Graceful Degradation of Services

3. Coarse-grain Parallelism

10/24/2014 6

C1. Crisp Interactive Response

Goal: Latency of infrastructure = tens of millisecond

10/24/2014 7

Face exists?

<700ms

Whose face?

~500ms

Voice Phrases?

~400ms

Humans are fast and sensitive

Satellite Phone?

ANNOYING!

S1. Crisp Interactive Response

Choice 1: standalone apps

Choice 2: offload to cloud

Choice 3: offload to cloudlets

10/24/2014 8

Distant Resources in Cloud

Nearby Cloudlet

Wi-Fi WAN

LAN

~74ms

<10ms

Glass Device

Exp. – Cloudlet Shortens Latency

10/24/2014 9

0

0.25

0.5

0.75

1

0 200 400 600

Response Time (ms)

Cloudlet

Cloud

Cloudlet vs. Cloud (Augmented Reality)

Offloading vs. Standalone (OCR)

Metric Standalone With Offload

Per-image speed (s) 10.49 1.28

Per-image energy(J) 12.84 1.14

Cloudlet shortens

response time

Offloading saves

latency and energy

C2. Graceful Degradation of Services

10/24/2014 10

What if offloading impossible?

Distant Resources in Cloud

Wi-Fi WAN

LAN

Situation 1: No cloudlet

Situation 2: No network

? ?

Nearby Cloudlet

Goal: still work during failures – with performance drop

Glass Device

S2. Graceful Degradation of Services

10/24/2014 11

Use fallback resources

No cloudlet – offload to cloud

No network – offload to carry-on device

Application-specific fidelity vs. Crispness & battery life

Distant Resources in Cloud

Glass Device

Nearby Cloudlet

Wi-Fi WAN

LAN

? ?

Wi-Fi or Bluetooth

On-body Offload Device

C3. Coarse-grain Parallelism

• Programming languages are different

• Runtime systems are different (different OSes, closed-

source, etc.)

10/24/2014 12

Don’t reinvent the wheel

Goal: reuse existing work, but…

Face recognition

Object detection

Activity inference

OCR ……

S3. Coarse-grain Parallelism

10/24/2014 13

VM Ensemble and PubSub Backbone

Core Algorithm

Logic

Virtual Machine

Dataset

Proxy

Config

Glass Device

S3. Coarse-grain Parallelism

10/24/2014 14

Cognitive Engine 1 e.g. Face recognition

Cognitive VMs

Video/Acc/

sensor streams

Device

Comm

UPnP

PubSub

Control VM

Cloudlet VM boundary

Sensor flows

Cognitive flows

User Guidance VM

VM Ensemble and PubSub Backbone

Cognitive Engine 2

Cognitive Engine 3

Cognitive Engine n

...

Wi-Fi

S3. Coarse-grain Parallelism

10/24/2014 15

Cognitive Engine 1 e.g. Face recognition

Cognitive VMs

Device

Comm

UPnP

PubSub Video/Acc/

sensor streams

Control VM

Cloudlet

User Guidance VM

Cognitive Engine 2

Cognitive Engine 3

Cognitive Engine n

...

Glass Device

Wi-Fi

Context Inference Wi-Fi

VM boundary

Sensor flows

Cognitive flows

VM Ensemble and PubSub Backbone

Exp. – Gabriel Overhead

10/24/2014 16

0

0.25

0.5

0.75

1

0 20 40 60 80 100

Response Time (ms)

Gabriel

Ideal

Echos every image

Dummy Cognitive Engine

No VM, echos from host

Gabriel Latency: ~50ms

Gabriel Overhead: ~3ms

10/24/2014 17

GDK Preview

Ice pack to cool down Glass

TCP Connection

Speech Guidance

Back-end Server

Prototype

10/24/2014 18

Cloudlet: 4 advanced desktop machines Running OpenStack – Virtualized Cloud Computing Platform

Back-end Server

Prototype

10/24/2014 19

Face Recognition

Object Recognition (1. MOPED 2. STF )

OCR (1. Tesseract 2. VeryPDF )

Motion Classifier

Augmented Reality

Activity Detection

Commercial Product

Based on Accelerometer

Prototype Cognitive Engines

Exp. – Full System Performance

10/24/2014 20

Gabriel architecture allows easy upgrade.

Cognitive Engines are slower

Cognitive Engine FPS Response time (ms)

Glass Life 1% 10% 50% 90% 99%

Face Recognition 4.4 196 389 659 929 1175

~1 hour

Object (MOPED) 1.6 877 962 1207 1647 2118

Object (STF) 0.4 4202 4371 4609 5055 5684

OCR (Open) 14.4 29 41 87 147 511

OCR (Comm) 2.3 394 435 522 653 1021

Motion Classifier 14.0 126 152 199 260 649

Augmented Reality 14.1 48 72 126 192 498

Exp. – Full System Performance

10/24/2014 21

Gabriel uses two-level token-based flow control

Cognitive Engines require different FPS

Cognitive Engine FPS Response time (ms)

Glass Life 1% 10% 50% 90% 99%

Face Recognition 4.4 196 389 659 929 1175

~1 hour

Object (MOPED) 1.6 877 962 1207 1647 2118

Object (STF) 0.4 4202 4371 4609 5055 5684

OCR (Open) 14.4 29 41 87 147 511

OCR (Comm) 2.3 394 435 522 653 1021

Motion Classifier 14.0 126 152 199 260 649

Augmented Reality 14.1 48 72 126 192 498

10/24/2014 22

drop

M A X ••

•

drop

drop

• •

•

Glass Device

Control VM

Cognitive VM

Cognitive VM

sensor

sensor data counts

yes no

a > b?

sum

a b

Processed Items Count

Drop Count

tokens

More in the Paper

1. Token-based flow control improves response

time a lot

2. Gabriel supports multi-VM parallelism

3. Tradeoff between fidelity reduction and crisp

user interaction

10/24/2014 23

10/24/2014 24

Conclusion & Future Work

Algorithms

Offloading Technology

Wearable Hardware

Gabriel: low-latency, flexible architecture

Speed improvement

needed

Cloudlets are helpful,

need good biz. model

Longer battery, better

thermal dissipation

10/24/2014 25

Cognitive Assistance Wearable Towards

Gabriel:

Contact: Kiryong Ha (krha@cmu.edu) Zhuo Chen(zhuoc@cs.cmu.edu)

Backup

10/24/2014 26

S3. Coarse-grain Parallelism

10/24/2014 27

Cognitive Engine 1 e.g. Face recognition

Cognitive VMs

Device

Comm

UPnP

PubSub Video/Acc/

sensor streams

Control VM

Cloudlet

User Guidance VM

Cognitive Engine 2

Cognitive Engine 3

Cognitive Engine n

...

Glass Device

Wi-Fi

Context Inference

VM boundary

Sensor flows

Cognitive flows

VM Ensemble and PubSub Backbone

10/24/2014 28

Prototype Mitigating Queueing Delay

Queues hurt latency:

Node A

Node B

Two-level token-based flow control

10/24/2014 29

Queues hurt latency:

Node A

Node B

Two-level token-based flow control

Throughput vs. Latency

= 1 0 ACK

Prototype Mitigating Queueing Delay

Exp. – Queuing Delay Mitigation

10/24/2014 30

0

10

20

30

40

0 100 200 300 400

Res

po

nse

del

ay

(s)

Request time (s)

token=1 token=2 TCP

0

200

400

600

token = 1 token = 2 tcp

nu

mb

er o

f fr

am

es

On-time Late

Dropped at Backend Dropped at Glass

Fidelity vs. Response Time

10/24/2014 31

S1. Crisp Interactive Response

10/24/2014 32

Justification of offloading

Table 1: Evolution of hardware performance

Table 2: Experiment result with OCR

Scale-Out of Cognitive Engines

• Cognitive Engine: Motion Classifier

– Slaves run in separate VMs

• Feature extraction

10/24/2014

• Classification

– A subset of classes for

each slave

• Waving

• Clapping

• …

33

Scale-Out of Cognitive Engines

# of slaves 1 2 4

Frame Rate / fps 9.8 15.9 19.0

10/24/2014 34