Social inclusion via augmented senses: from academia to ... · Social inclusion via augmented...

Social inclusion via augmented senses: from academia to society

Laura Giarré CSAIL- MIT 29/4/2015 1 L. Giarré, University of Palermo

The Blind Leading the Blind, P. Brugeul the elder. 1568, Napoli Capoimonte musuem

CSAIL- MIT 29/4/2015 L. Giarré, University of Palermo 2

Aknowlegement


Andrea Bocelli, Veronica Berti, Laura Biancalani, Olimpia Angeletti, …

Ilenia Tinnirello, Domenico Garlisi, Daniele Croce, Fabrizio Giuliano, Natale Galioto, Benedetta Falmi, Claudio Bellotti, Pierluigi Gallo, …

Seth Teller, Munther Dahleh, Paul Parravano, Daniela Rus, Anantha Chandrakasan, Nick Wang, Robert Katzschmann, Jordan Allspaw, Rahul Namdev, Dongsunk Jeon, Santani Teng, JJ, Nathan Ickes, ...

Franco Lisi, Barbara Leporini, Antonio Quatraro, Giuseppe Scaccia, Luca Ciaffoni , …

Aknowlegement

• How much the empowerment of people with specific needs can benefit from innovative technological solutions?

• Is there a future for assistive technology devices outside the Academic Laboratories?

• What are the steps to design a system that can be easily accepted ?

• Are the investors ready to put an effort into assistive technology?


Open questions

• Social innovation means developing new ideas, services and models to better address social issues.

• Social investment is about investing in people

• Challenge: to strengthen people’s current and future capacities, and improve their opportunities to participate in society and the labour market.


Social Innovation


Impact on society

Font: MOMA Jacob Lawrence. The Migration Series.

Interdisciplinary approach Integration of many different areas


Technology Societal Problem

Psycology/ cognitive

Social inclusion: is Technology a solution?

Design


Assistive Technology

Integration&Interdisciplinarity

Can technology help a Visually Impaired Person

• to live a more independent life, working and socializing

• to move autonomously and safely in an unknown environment

• to retrieve information about the places nearby

• to recognize the persons they are interacting with and understand their feelings

• to perform day life activities with increased autonomy


Challenges for VIP


S

Machine Learning Filtering

Signal Detection Feature Extraction Computer vision

Haptic Interface Vibration Braille Pad Music Audio signal Wearable vibrating bracelet/belts

Camera Sonar/laser Acceleremoter RFID IMU iBeacon

CYBER SIGHT SENSE

Sensors Actuators

Computation

Solution

Invasive solutions:

to directly stimulate the deficient sensory sistem

– Implanted neuro-prostheses relying on the integrity of visual pathways in the brain regions

– The previous visual experience is a pre-requisite to trigger and interpret the visual perception


State of the art

CSAIL- MIT 29/4/2015 12 L. Giarré, University of Palermo

• Use of one or more SPARED sensory modality to supply information normally gathered by the impaired sense • Aids employed regardless of the different clinical features of blindness •Basic Examples:

•Braille reading: fluently read through raised patterns • Long cane: uses tactile information to explore the surroundings

Non invasive Solutions


To support sensory deprived individuals in their daily activites

First step : General Purpose SSDs

40 years of research with the goal of achieving tactile vision substitution, moving from larger displays on the back and admonen to much smaller displays on the tongue the stream of images from a video camera converted into stream of tactile or auditory images.

Assumption on perceptual learning: people eventually is

able to process the tactile stimuli and automatically perceive relevant features of the surroundings.

Sensory Substitution Devices (SSDs)


•Early success with simplified tasks and conditions has bred optimism about eventual success in performing complex tasks in real-world conditions •Walking through environment cluttered with small low contrast obstacles

Fallacy of the First Step


• Identify what type of information and how much of that information is necessary to perform a GIVEN task. •Characterize the sensory substitution by their spatial BANDWIDTH ( the capacity of conveying spatial information) • Is the substitution adeqaute to handle this information? •SUCCESS CASES: Long white cane ; Braille note GPS

Provision of reliable and easy to access to task- critical information while excluding other environmental info

Second Step: Special Purpose SSDs

• Miniaturization of actuators and

electronics : new systems embedded into clothing, devices

worn on the finger, hands, wrist, forearm, tongue, head, chest, abdomen and feet

• Hearing and touch is the first and second major senses, and it gather enough information from the environment to perform daily tasks.

• Assistive devices provide acoustical and tactile feedback to compensate for visual information.


Wearable Solutions


Evolution (safe mobility)

a) Inspired by bats and designed using microprocessor and ultrasonic devices b) A user pushes the GuideCane, which detects an obstacles : a walkable direction of motion c) A wearable device: distant objects detected, visual information detected into tactile

signals d) RGBD camera based system: visual odometry, navigation, and a path planner to build a

voxel map and analyzes the traversability; audio and haptic feedback. (e) Mit fifth sense system aimed at delivering high-level information via off-the-shell soft

neck and body-worn haptic array.


Evolution (reading)

EyeRing: a finger worn camera module and wirelessly connected smartphone, aural feedback on objects captured by the lens

OrCAm: a smart camera mounted on the frames of the eyeglasses. It recognizes text and products, and speaks through a bone-conduction earpiece.

Font:Eyering, MIT Media Lab

Font: www.orcam.com


Evolution (face & expression recognition)

.

Detect who the person is and distinguish the person’s moods: happy, sad, neutral, angry or surprised emotions on the face of the person they are interacting with.

Font: Metro/Handout/Gagandeep DulaySamer Toukan and Easa Ahmadzai displaying the project at the University of Windsor


Evolution (use of remote assistant?)

.

A Network of Eyes: Be My Eyes is an app that connects blind people with volunteer helpers from around the world via live video chat.

Blind Requests Assistance

Volunteer Receives Video And Describes


• Understand the reasons behind the limited diffusion of SSDs among the blind communities. • Issues to be considered in designing future devices:

•Ergonomic/functionalities •Neurocognitive •Psychological

Future for SSDs outside Laboratories?


• Small in size • Hardware robust but light • Optimized Battery life • Appeal of exterior design • Cosmetic acceptability is more important than technological potential benefits • Functionality: TRANSMIT INFORMATION to the remaining sense without INTERFERING

Ergonomic


COGNITIVE ERGONOMICS: direct mapping of the visual environment into ‘one2one’ auditory or tactile information is likely to result too complex

If auditory or tactile stimulation produce ‘cognitive overload’, since the spatial bandwidth largely exceeds that of the other senses

Efficient and intuitive ways to map different modalities

Fundamental question: Amount of information that should be selected and pre-processed

Neurocognitive


•The blind community is opting for a status quo

REASON: the degree of real and perceived complexity in understanding and using SSDs

•Innovation is a learning process where the reserachers developing the new technology need to involve the people who might benefit from it

EMBED THEIR EXPERIENCES IN THE DESIGN

•Visually impaired persons benefit from training program: importance of the learning phase

Accessibility and Training


• What is the easiest way to let blind people use the SSD:

– minimize the external infrastructure

– use their own smartphone

– develop cheap (computationally) algorithms running on smartphones

– make use of all the sensors already embedded in the phones

A case study:


A case study:

The Problem

Incentivizing autonomous mobility

of visually impaired people

– Accompanying the user along the path step-by-step

– Giving directions towards the destination

L. Giarré, University of Palermo 27

Existing solutions: Personal Assistant, Tactile Paving GPS or other indoor positioning systems

CSAIL- MIT 29/4/2015

The Innovative Solution

28 L. Giarré, University of Palermo

• Unmet needs:

– autonomy, information on destinations, limited interaction with the environment

– User interfaces for blind people, continuous reference

• Novel technological solution for unmet needs! – From mute paths to talking paths

– ICT for improving user interactions (robotic&vision)

– Haptic interfaces


1. Smartphone camera continuously captures the scene in front of the person.

2. The algorithm identifies the path by recognizing colored strips painted or sticked to the floor.

3. The users receive a vibration signal through his cell phone allowing to understand the direction to follow .

4. It interacts with the location tags to extend the information related to the position.

CSAIL- MIT 29/4/2015 29

Occasionally arianna can use audio for special ad hoc information

Camera view

Arianna application

identifies the strip

HOW?

L. Giarré, University of Palermo

CSAIL- MIT 29/4/2015 30

Tape and QRcode laying

User arianna app installation

Map creator and

server configuration

o app configuration

System usage

Installation step:

1) Tape laying

2) Map creator

3) Map selection

4) System usage

What is needed


Design

31 L. Giarré, University of Palermo

• HW: Small in size , robust but light, cosmetic acceptability commercial smartphone • INTERFACE: using vibration to not interfere with auditory system ; audio only if necessary •COGNITIVE ERGONOMICS: no overload. Follow a line step by step •EMBED blind EXPERIENCES IN THE NEW TECHNOLOGY


Competitive Advantages

• Technology:

– From computer-vision for robot navigation to computer-vision for visually impaired people

– Haptic interfaces for guiding the user

• Added value:

– flexible paths, interactive and extensible system

– Deployment much easier than tactile paving

• Durability:

– Highly-tailored

– unlikely general-purpose IPS offering similar services L. Giarré, University of Palermo 32 CSAIL- MIT 29/4/2015

State of the art on indoor navigation

L. Giarré, University of Palermo 33 CSAIL- MIT 29/4/2015

• Indoor navigation: – Triangulation of RF signals (WiFi)

– Direct sensing (RFIDs, ultrasound, Bluetooth, iBeaconn,...)

– Pattern matching

– Dead-reckoning (accelerometers, gyroscopes, …)

• Often dedicated hardware is required – Expensive devices, complex deployment

• Low accuracy (sometimes <5m) – Insufficient for navigation of the visually impaired

• Audio feedback is annoying!!!

Inside 1/3


Computer vision algorithms •Follow “the line” with Computer Vision

•Hough transform •Find lines in image

•Canny •Find edges in image

•Good computer vision algorithms are expensive • What about running on smartphones?


Inside 2/3

Computationally Cheap algorithms • Use HSV (hue-saturation-value) systems: robust to ambient light changes (more than RGB) • Implement a “simple” range detector:

• Find all the pixels in the image between specific tuples


Computationally Cheap algorithms

Inside 3/3

• Single color recognition fails: Always • Idea: use two “known” nearby colors o Use a line with two stipes o Find both colors o No complex matrix operations: only “if”s

• Robust to camera noise: can’t go wrong (mostly) if properly implemented • Indoor and Outdoor

Future of 1/4


EKF- based tracking to predict user movements

Human in the loop to calibrate the feedback

Assumptions – Smartphone and user positions are the

same – Movements in 2D only – “Constant” speed (changes predicted

by the EKF)

• Two corrections proportional to – Heading deviation from the path (β-α) – Distance from the path (d) and time to

recover (Tx)

Future of 2/4


• EKF and Human in the loop

• m and n model the human behavior

• represent the human perception of the signals

• is a costant angle (positive/negative if the path is lost on the left/right)

Future of 3/4


• EKF and Human in the loop

Future of 4/4


• Optical flow (Pedestrian navigation using IMU+vision)

Font: Botterell et al, 2009

Challenge: trade-off performace Vs. complexity


How to evaluate the system?

• Run specific tests: decide a task (a path presenting curves, crossing points, beginning and end, Qr codes)

• Define some mertrics: time, number of QR codes missed, etc (quantitative)

• Run follow up with questionnaires (qualitiative)

• Importance of feedback from Blind Communities!!!

Questionnaires


Tests with Blind Communities

• Istituto ciechi Milano , Bologna

• Unione Ciechi Palermo, Firenze

• Demos during expos

• We collected subjective and objective data and feedbacks.

• A permanent installation at the University Campus of Palermo. This will be used for more complicated tests of the product and for scientific purposes.



The Market (B2B)

Airports

Offices/Mall

Sidewalks

Museums

Musuem installation


CSAIL- MIT 29/4/2015 46

www.turismomassello.net: a mountain route, located in Pinerolo (CN) Installation of a system arianna in the tourist trail; Test phase of the system; Customizing phase of the system for the tourist

routes considered (including point of interests descriptions and audio-guide);

Simple tape

2 tapes path

Tourist route installation


http://www.turismomassello.net/

http://www.turismomassello.net/


Learning through games: From To

•Training to the use of ARIANNA •Interactive control using audio vibration for augmented reality orientation. •Learning through Amusement

• Educational: Allocentric vs. Egocentric perception

– You win if you follow the line in the shortest time

– You get scores if you pick up thinks (reading the QR code along the path)

– More scores if, upon requests, you find the short cuts


Learning through games: From To

• The investors have not responded very well: A Too niche market? No possible revenues?

• In.Sight with Arianna has won Startcup Palermo, UK Trade&investiments Award

• PNI 2014, Smart City Innovation, Italian Venture Torino, Shenzen Design Exhibition


The Investors?


Conclusions 1: Step for the Design

• Sensory overload X

• Long learning/training time X

• Acoustical feedback X

• Wearable, light, invisible √

• Tactile feedback √

• Acceptance √


Conclusions 2: Feedback

• Blind community: To be involved in the design and tests of the prototype/product

• Run Extensive tests to evaluate the effectiveness: collect objective and subjective data


Conclusions 3: Go to Market

• Find Investors

• Find Government Grants

• Find Commercial Partners

• Strongly believe in the use of accessible technology for a better society and social inclusion

• Disseminate ideas and experience!

Thanks to


Social inclusion via augmented senses: from academia to ... · Social inclusion via augmented...

Documents

Transcript of Social inclusion via augmented senses: from academia to ... · Social inclusion via augmented...