Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving

through Cloud Computing

Vladimir Kulyukin, Tanwir Zaman Abhishek Andhavarapu, Aliasgar Kutiyanawala

Department of Computer ScienceUtah State University

Logan, UT, USA

http://www.linkedin.com/pub/vladimir-kulyukin/23/2a2/150


http://www.icchp.org/

Big Picture

Independent Shopping for Visually Impaired (VI) and Blind Individuals



Why Is Blind Shopping Difficult?A modern supermarket has a median area of 4 300 m2, stocks an average of 38 718 products, and services approximately 600 customers per hour (6 000 to 7 000 per day)



Hardware Evolution2003-2005

2006-2008

2008-2010

2010 -Now

RoboCart ShopTalk ShopMobile



ShopMobileComputer Vision + Camera Alignment

= Independent Eyes-Free Store Browsing on Smartphones



Parallel Threads of Computation

Barcode Scanning Camera Alignment



Automated Approaches: Problems & Alternatives

● Two problems with automated approaches:– Error-prone: false positives and false

negatives– Greedy power consumption

● Two alternatives: – Remote caregiving – Crowdsourcing

● Both can be used to augment automated approaches



Remote Caregiving vs. Crowdsourcing● Crowdsourcing may not be suitable for time

sensitive tasks and tasks that require mutual trust● Time sensitivity can be addressed through

increased helper volume (unlikely to materialize for smaller disabled populations)

● Trust may be a more serious issue for crowdsourcing

● Remote caregiving addresses trust but requires dedicated caregivers



Eyesight Sharing & Cloud Computing ● Cloud computing infrastructures and real-time

video streaming protocols make it possible for sighted individuals to share their sight with their VI friends remotely

● Hearing can also be shared remotely● Amount of caregiving can be dynamically adjusted

at run time



TeleShop

Blind Shopper

Caregiver

Caregiver

Wi-Fi/3G/4G

Blind Shopper



TeleShop Architecture

Caregiver Blind Shopper



A Cloud-Based Caregiving Architecture

Client Client




Client Client

Caregiver




Client

Amazon EC2Elastic Computing

Service

Client

Caregiver


Service




Client


Service

Client

Caregiver


Service

Android C2DMCloud 2 Device

Messaging




Request Client


Service

Client

Caregiver


Service


Messaging




Request

Notification

Client


Service

Client

Caregiver


Service


Messaging




Request

Notification

Notification

Client


Service

Client

Caregiver


Service


Messaging




Request

Notification

Notification

Help

Client


Service

Client

Caregiver


Service


Messaging




Request

Notification

Notification

Help

Help

Client


Service

Client

Caregiver


Service


Messaging




Request

Notification

Notification

Help

Help

Client


Service

Client

Caregiver


Service


Messaging

Client

Client

Client

Client

Caregiver

Caregiver

Caregiver

Caregiver



Clients & Caregivers

● Clients send product images (video streams are possible but consume data plans fast)

● Caregivers look at images, speak product names (type if SR does not work), send text back to clients

● When caregivers cannot identify products from received images, they can request a new image



Cloud-Based Image Matching

● SURF ran in the cloud was used as a black box image matching algorithm

● SURF was used to speed up caregiver's product recognition

● SURF returned top n (n=5 in our case) images and caregiver would verify the correct product name, if it is in top n, or speak/type the correct name, if it is not



Three Experiments

● A laboratory study with a blindfolded subject● Two field studies with a blindfolded subject

at Fresh Market, a supermarket in Logan, UT● A field study with a blind subject was

completed at Fresh Market, a supermarket in Logan, UT, after the paper was accepted at ICCHP



Lab Study ● 20 products (boxes, bottles, cans)● SURF trained on 100 images (5 images per

product)● Blindfolded subject was in a lab; caregiver was in a

different room● Subject given one product at at time and asked to

recognize each● Subject and caregiver used Google Nexus One with

Android 2.3.3● Data link was Wi-Fi



Supermarket Experiments ● Setting was at Fresh Market, a supermarket in

Logan, UT● 45 products (boxes, bottles, cans) from 9 aisles● SURF was trained on 370 images● Blindfolded subject used Galaxy S2 with Android

2.3.6 was at Fresh Market● Caregiver used Google Nexus One with Android

2.3.6 was in a different building, a mile away from the supermarket

● Data line was 4G



Supermarket Experiments

● In experiment 1, subject was given 16 products by assistant, one product at a time

● In experiment 1, SURF was turned on● In experiment 2, subject was given 17 products by

assistant, one product at a time● In experiment 2, SURF was turned off, because of

poor performance in experiment 1



Experimental Results

Environment # Products Mean Time STD TOP 5 Mean SR SR Fails

Lab Study 16 40 .00021 8 1.1 0

Store 1 16 60 .00033 0 1.2 2

Store 2 17 60 .00081 0 1.1 3



Conclusions ● SR appears to be a viable option for

product naming and tagging on Android phones

● Poor SURF performance was probably due to our limited understanding of its parameters

● Basic trade-off: battery life vs. data plan consumption



Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Technology

Transcript of Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing