A context aware and proactive in-vehicle information system:

Matching infotainment with safety

Luca Contini - Akhela 10/10/2013

Agenda

• SOTA and current limitations

• The Solution

• Car Sensors

• Software Reference Architecture

• Aggregator

• Context Engine

• Proactive Decision Engine

• Adaptive HMI Engine

• Database

• Results

• Future Work

SOTA

Current IVI systems are difficult to navigate: user must dig into too many levels

SOTA

Information is not filtered or recommended proactively

SOTA

The interface can be distractive and unsafe

The Solution

A minimal, context sensitive and proactive user interface

The Solution

Featuring a warning-based recommendation system

The Solution

To achieve context modelling and thus proactivity goals,

sensors must be used

Car Sensors

Visual Odometry

Visual Search

Weather Distance to destination

Cruise Range

Virtual Sensors

OBD Fuel Level

OBD Sensors

OBD RPM

OBD Engine Temperature

OBD Speed

Local and remote database

Proactive HMI

External cameras for video processing

Weather

Visual algorithm dedicated HW

Location sensor

Car Sensors

Visual Odometry

Visual Search

Weather

Distance to destination Cruise Range

OBD Fuel Level

OBD RPM

OBD Engine Temperature

OBD Speed

Location sensor

Car Sensors

Visual Odometry

Visual Search

Weather

Distance to destination Cruise Range

OBD Fuel Level

OBD RPM

OBD Engine Temperature

OBD Speed

Location sensor

CONTEXT DEFINITION

PROACTIVITY

Car Sensors

Visual Odometry

Visual Search

Weather

Distance to destination

Cruise Range

OBD Fuel Level

OBD RPM

OBD Engine Temperature

OBD Speed

Location sensor CONTEXT

DEFINITION

PROACTIVITY

Software Reference Architecture

Visual Odometry

Visual Search

Weather

Distance to destination

Cruise Range

OBD Fuel Level

OBD RPM

OBD Engine Temperature

OBD Speed

Location sensor CONTEXT

ENGINE

Reference Architecture

AGGREGATOR

ADAPTIVE HMI ENGINE

PROACTIVE DECISION ENGINE

To achieve context definitition and proactivity we use a 4 layer

software stack

Reference Architecture: Aggregator

Cruise Range

OBD Fuel Level

OBD Speed Location sensor

AGGREGATOR

CONTEXT ENGINE

The Aggregator component collects sensor data, aggregates them

according to specific rules, and pushes them to the context engine

Reference Architecture: Context Engine

Ontology Rules

CONTEXT ENGINE

Context Facts

m_ContextAction0= "Show Default Panel"

m_ContextAction1= "Show Parking Warning"

m_ContextAction2= "Show Fuel Warning"

m_ContextAction3= "Show POI Available Warning"

m_ContextAction4= "Show VS Match Warning"

PROACTIVE DECISION ENGINE

The Context Engine applies the ontology rules to the sensor

values and generates lists of facts

Reference Architecture: Proactive Decision Engine

The Proactive Decision Engine is composed by separated sub-

engines communicating via task-board

POI Manager

Panel Manager

Navigation Manager

Warning Manager

Taskboard

Proactive Decision Engine

Adaptive HMI Engine

Reference Architecture: Proactive Decision Engine

Each sub-engine filters its specific category of facts and creates and

ordered (priority based) list of facts (actions) to be shown in the HMI

Adaptive HMI Engine

PDE takes “decisions” between possible

solutions

POI Manager

Panel Manager

Navigation Manager

Warning Manager

Proactive Decision Engine

Show default panel

Show fuel warning

Reference Architecture: Adaptive HMI Engine

Finally, the Adaptive HMi Engine selects the proper modality

Adaptive HMI Engine

PDE takes “decisions”

AHE selects modality

Show default panel

Show fuel warning

POI Manager

Panel Manager

Navigation Manager

Warning Manager

Proactive Decision Engine

Show default panel

Show fuel warning

Database

Remote database manages:

• Normal Points of Interest

• Visual Search

• Multimedia information ofr Augmented Reality

The database is locally buffered when

a specific route is selected, to avoid

connection issues during the trip

Results

The result is an HMI proactively presenting the information to the driver

Results

When a route is not set, the system calculates the cruise range based on current fuel level and shows it on the map

Results

When the fuel level is low, the system recommends the closest gas stations

Results

3D Bubbles are used for Augmented Reality trip preview

Results

Visual Odometry algorithms are used when the GPS signal gets lost in urban canyons, keeping the car position on the map up to date

Results

Visual Search algorithms find a visual match on what the camera is

shooting, allowing specific POI information to be delivered when

actually facing a meaningful building

Results

Augmented reality is used only when the car is stopped

Results

Safety is achieved by reducing the amount of information when dangerous or not needed

Future Work

• Improve Context Models and Rules/Engine

• Improve Proactive Decision Rules/Engines

• Improve mental workload control

• Improve user state detection

Thank You