Demola smart cabs_20120502

SMART CABS:

MACHINES THAT

KNOW THEIR

DRIVERS Rod Walsh, Petri Murtomaki, and

Kimmo Vänni

TAMK

Demola InnoSummer 2012 version date author details

0.1 15.03.2012 RW & KV Created & first ideas

0.2 16.03.2012 Rod Walsh Minor improvements

0.3 02.05.2012 Rod Walsh Filled out the “complete story”

© TAMK, 2012 . ALL R IGHTS RESERVED. TAMK CONFIDENTIAL . 1

COMING UP IN THIS SLIDE SET…

Why: Better Performance in Forestry

What: The Human Touch

How: The Demo

Where: The Big Idea

Approach: Approach


BETTER PERFORMANCE IN FORESTRY

The commercial performance of large human-operated machines is

largely determined by the performance of the human operator

Today, human operator performance is largely driven by hard external

factors, such as training, experience and attitude

Dynamic factors are “left to care for themselves”: such as tiredness,

alertness, attentiveness, happiness, etc.

But we want to use technology and human-insight to monitor these soft

internal factors

And improve working life, long-term health and commercial productivity


THE HUMAN TOUCH We will take a look at the emotional state-of-mind of

operators using face, sound and posture monitoring technology with pattern recognition

And use our knowledge of these soft internal factors for improvements:

Happier and lower-stress work (short and long term benefit for the employee)

Better productivity (short and long term benefit for the employer)

By:

Dynamically modifying the working environment for the better (short term)

Identifying positive patterns of emotion affect on human performance & motivation, and then matching practices, assignments and environments the patterns (long-term)


state of mind

Pattern

recognition

Simple changes

• Music, lighting, airflow, …

Working

practices

non-contact sensing

Psychology

& processing

THE DEMO

Multiple HD webcams, microphones and PrimeSense IR sensors (e.g. Kinect) will be arranged to monitor a human “operator” (non-contact sensing)

(For versatility, an “office desk operator” setup is needed. The team may take physical forestry machine mock-ups and closeness-to-reality to higher levels.)

A set of “states of mind” that are relevant to machine operator performance and wellbeing will be selected

Quickly selected emotions at first (for rapid development) & then iterated

Sensor signals are classified for the “states of mind”

Classifier(s) will be “trained” and tested. Training and testing will begin with “acted emotions” and tightly iterated between the pattern recognition and the pyschology/emotional model.

Offline: all sensor and analytics data will be logged, to allow discovery of longer-term patterns (such as time of day patterns)

Real-time: The instantiations state-of-mind is matched against a “task model” and need for corrective action (on the operator) is calculated

As determined, corrective action is taken to change the operator’s environment

The effects and affects are logged to determine whether the action succeeded

(The “office desk simulator” can be a PC display simulation, or better…)

SEE NEXT SLIDE FOR VISUAL DESCRIPTION


Examples of corrective action:

• Encouragement

• Stimulation

• Pause/end of task

• Verify the measurement

Examples of Job improvements:

• Productivity

• Volume

• Errors

• Motivation for the job

• Intervention before

problems become critical

Examples of “state of mind”

• Tiredness

• Boredom

• Willingness to work

• Fear/anxiety

• Happiness

Database:

state of mind log

sensor logs

non-contact sensing:

video, image, audio

7/10 capability

~7/10 capability

state of mind

estimation

Match

with

task

8/10 minimum

Pattern

recognition

Simulate simple changes


real-time

logged

offline logged


TH

E D

EM

O

THE BIG PICTURE


For the long-term benefits, the data can be used to change the design of

working environments and practices, so…

The demo would be integrated to a larger system (see next slide)

Existing telematics data from the forestry machines can introduced to

the common database and analyzed for patterns between operator

state of mind and machine behavior (for further insights and causalities)

This is beyond what the team needs to do!

The team’s innovation and excitement decide what is done beyond the core demo

Database:

telematics log

state of mind log

sensor logs

capability

capability

state of mind

Match

required

Logging telematics

(exists already)

Human impact

on work

quality &

productivity &

machine

performance

Machine +

environment

impact on

human

operator


corrective action

Improvements

THE BIG PICTURE

APPROACH


In theory, the team is free to adopt any approach that:

Works well, looks great and receives “ooh” and “wow” sounds

Fits the objectives

Is reusable, extendable and portable (as a whole and as components)

Meeting these needs in one go is near impossible, so iteration, communication

and sharing are critical – and at high speed!

In practice, the support team has some useful experience and advice:

Short design, implementation and demo iterations are the safest and coolest

Stick to technologies which are cross-platform and open (when possible):

E.g. HTML5, OpenNI, Published solutions, etc. as applicable

We will supply USB webcams (inc. microphones) and PrimeSense IR sensors

Code should be runnable on Mac/Win/Linux (Ubuntu is our favorite Linux)

We will workshop together to best use the team’s and the support team’s knowledge

…

Some support slides

SMART CAB +

AFFECTIVE ROBOTS


Together with another awesome project,

we could close the loop on emotional

feedback (possible project extension)

What setting or stage would unlock,

emphasize or inhibit which affects?

modeled human-like emotion Full

body face

Emotive

commands

like:

• be happy

• welcome

• Reject

• cry

actual robot virtual robot

state of mind

estimation

“corrective”

action using and

emotionally-savvy

avatar

state of mind

1. Perform the emotion

2. Perform for the emotion

3. Read/write emotion?

Match

Goal

Database:

telematics log

state of mind log

sensor logs

non-contact sensing:

video, image, audio

7/10 capability

~7/10 capability

state of mind

estimation

“state of mind”

Match

with

task

8/10 minimum

Logging telematics

(exists already)

Human impact

on work

quality &

productivity &

machine

performance

Machine +

environment

impact on

human

operator

Pattern

recognition

Simulate simple changes


real-time

logged

offline logged


corrective action

Job improvements

SIMPLE ONE-SLIDER


telematics

Design of

practices &

environment

Demola smart cabs_20120502

Education

Transcript of Demola smart cabs_20120502