Accessible Shopping With Innovative Hardware and

Usable ApplicationsMichelle McManus

Presentation Overview

• This presentation is to bring your attention to a research project which is occurring at multiple universities. The goal of this project is to develop adaptive technology which will help blind and visually impaired people be more independent in their daily lives.

Background Information

• Before discussing the current project. Here are some pieces of technology which were proposed to accomplish the task of helping blind/visually impaired people shop or be more independent.

RoboCart

• Description: This was a custom shopping cart that was built on a laser guidance system and RFID tags. The laser guidance system allowed the cart to perform collision detection, as well as help the user of the cart navigate around the store. Then, using a RFID tag system, the shopping cart can stop and alert the user when they are near an item on that person’s shopping list.

RoboCart Problems

• As one can see this product would be hard to implement for several reasons. One, it is big and bulky.• Two, this product could only be set up for use in one store and the

store would need to add RFID tags to all of their products. A lot of work for very few users!

Guidecane

• Description: This was a piece of technology built off of the standard white cane. Researchers attached a wheel and a motor to the bottom end of the cane, which allowed the device to sense for obstacles in front of it, and then re-direct the person using the cane if obstacles were detected.

Guidecane Problems

• From the way this device looks, it does not seem very useful to me. I’m not sure what it could detect differently than the original white cane. I don’t think it is or would have been cost effective.

SpaceSense

• Description: This was based on a set of 9 haptic motors being attached to the back of an iPhone. The motors were arranged in 3 rows of 3, with the intention of providing directions via these motors in outdoor spaces.

SpaceSense Problems

• This product looks a bit cumbersome even though it is using an IPhone. I do not understand how a person could perceive all of the information using this many motors.

MindMesh

• Description: This is a highly experimental technology developed by Dr. Steve Mann at the University of Toronto. He has dictated a great deal of development in this field. The device itself is rather extreme. It features a large number of electrodes placed on top of the head. The ultimate goal is to be able to translate video data directly to the brain to bypass vision all-together.

MindMesh Problems

• This device is the most extreme. I can’t imagine anyone wanting to where this in public at least from what I’ve been told. The way it has been described sounds as if it would actually work better for people who have learning disabilities.

The Third Eye Project

• The URL for the Third Eye Project: http://www.cse.psu.edu/research/visualcortexonsilicon.expedition/description.html

Third Eye Project Scope

• The Third Eye Project has multiple research teams at many universities. Penn State is only one of a group of ten or so universities which include:• Massachusetts Institute of Technology• University of Southern California• York College• Among others

Third Eye – Broader Impact

• Machine vision systems that understand and interact with their environment in ways similar to humans will enable new transformative applications. The project will develop experimental platforms to: (1) assist visually impaired people; (2) enhance driver attention; and (3) augment reality to provide enhanced experience for retail shopping or a vacation visit, and enhanced safety for critical public infrastructure. This project will result in education and research artifacts that will be disseminated widely through a web portal and via online lecture delivery. The resulting artifacts and prototypes will enhance successful ongoing outreach programs to under-represented minorities and the general public, such as museum exhibits, science fairs, and a summer camp aimed at K-12 students. It will also spur similar new outreach efforts at other partner locations. The project will help identify and develop course material and projects directed at instilling interest in computing fields for students in four-year colleges. Partnerships with two Hispanic serving institutes, industry, national labs and international projects are also planned.

Third Eye – Penn State

• Penn States portion of the project is to develop a device that will detect and relay product information in a grocery store to blind and visually impaired users. The researchers are going about this project by collecting data and feedback from potential users from the beginning. They have developed a database of people to assist with ongoing research and production of the product.

Third Eye – Penn State Prototype

• The first prototype of the device is a glove that has four motors; one for left, right, up, and down. The glove will connected to a Surface Pro 3 which will scan store shelves using computer vision techniques and give haptic feedback to the user through the glove.

Penn State – Glove Prototype• The image to the right shows the current

version 1.0 of the haptic glove currently being tested

• It is controlled by an Arduino board which is wired to four motors

Third Eye - Prototype

• This device is an ongoing project that will be developed and refind over a 5 year period. The intent is to make the device as small and as portable as possible along with being user friendly allowing the user to receive as much information during a shopping experience as they desire.

Third Eye - Funding

• The project is being funded in part by a grant from the National Science Foundation (NSA) and will go through a very intense review process each year. The research team at Penn State is lead by Dr. John M. Carroll and Dr. Mary Beth Rosson. The team is dedicated to creating a project with the end user in mind which is appreciated by the blind and visually impaired user.

Third Eye in the Media

• Article published in the Penn State research magazine. • URL: http://news.psu.edu/story/357788/2015/05/27/research/machine-

vision-system-could-help-visually-impaired-shop-food

• The article gives a nice description of the users as well as the research team and • Explains the process of the project.

Other Technology in the News – TapTapSee• TapTapSee is a mobile app for

the iPhone• It uses the built-in camera with

VoiceOver to give descriptions of objects photographed with the camera• Response time is typically a few

seconds, and it is incredibly accurate• Can recognize proper names of

items, as well as broad descriptions

Be My Eyes

• A smartphone-based app which connects blind individuals to connect with sighted individuals via a webcam• A volunteer “lends” their eyes

through the smartphone to help blind individual with numerous tasks• As of 6/18/2015, Be My Eyes had

232,383 sighted volunteers registered, and 19,969 blind users registered.

‘XploR’ Mobility Cane

• Uses smartphone technology incorporated into a traditional white cane to recognize faces from up to 10 meters away• Also includes GPS functionality to

assist with navigation• Still in development, but has been

featured recently on a number of news outlets

Third Eye - Contact

• For more information contact the research team at:• Dr. John M. Carroll – jcarroll@ist.psu.edu• Dr. Mary Beth Rosson – mrosson@ist.psu.edu• Jake Weidman – jyw5163@ist.psu.edu• Sooyeon Lee – sus131@ist.psu.edu

• Or me Michelle McManus• Mlh328@psu.edu