Image processing techniques for driver assistance

Razvan Itu June 2014, Technical University Cluj-Napoca

IntroductionComputer vision & image processing

• from wiki: “any form of signal processing for which the input is an image”

• computer vision - subfield of artificial intelligence in computer science

• sometimes referred as “the emulation of human vision by a machine”

Popular libraries and software• OpenCV - most popular library,

the internet is full of examples & tutorials, good documentation

• FastCV - library for mobile only, from Qualcomm the makers of Snapdragon processors found on most of today's smartphones

• matlab - also used, especially in the research field, provides one of the best camera calibration modules, very good to quickly prototype & test algorithms

ADAS

• Adaptive cruise control • Lane change assistance • Collision avoidance system

(pre-crash) • Traffic sign recognition • Vehicular communication

systems • Automatic braking, parking,

etc.

ADAS - Advanced Driver Assistance Systems Systems to help the driver in the driving process, examples:

Image source: Continental Automotive Systems

ADAS - Computer vision research2 main categories for ADAS using image processing: • single camera = monocular • stereo cameras = binocular (2 or

more cameras) some systems use more sensors in addition to the cameras sensors used to capture proximity/distance to objects: radar(RAdio Detection And Ranging) lidar(LIght Detection And Ranging) sonar(SOund Navigation And Ranging)

Image source: Texas Instruments

Image source: Bosch

Monocular• not so popular, single camera = cheap, maybe

improved speed/performance (you process half the number of frames/images compared to a stereo setup)

• reduced accuracy, one camera means no depth perception/knowledge

• mostly used to: detect lanes, detect traffic signs, maybe traffic light(s)?

Image source: SAE International

Stereo vision

• high accuracy, the 2 cameras mimic the human vision, the two eyes add a "depth" perception of the surroundings

• good usages in detection & tracking of the obstacles

• good accuracy because of the two camera setup that adds depth

mono vs stereo: in monocular approaches there are assumptions of a flat road, a constant pitch angle or absence of a roll angle

• most usages in automotive industry, expensive systems

• use special cameras that are calibrated & set-up for the specific hardware that is being used

Image source: Subaru EyeSight System

Stereo vision - continued• More on depth: you can get depth information from

a pair of images of same scene

• Create a disparity map

• Disparity refers to difference in image location of an object seen by the left & right eyes => in computer vision, disparity = the difference in coordinates of similar features within a pair of stereo images

Stereo vision - continued

Disparity map: Mars rover uses it to navigate through obstacles!

Image source: NASA

Basic algorithms & approaches1. Camera calibration

• using special software and by taking repeated photographs of a known pattern

• OpenCV provides a calibration software, most used is from the matlab

• through calibration process you get to know some of the camera parameters: intrinsic & extrinsic

Image source: dreamincode.net forumsImage source: http://www.driveassistapp.com

Basic algorithms & approaches!

2. Traffic sign recognition

• color based: for example search the RED color to detect the STOP sign

• shape based: detect the round shape of the SPEED LIMIT sign

• machine learning based: use a set of training images that contain both good & bad examples (ex: use Viola Jones detector)

• Viola Jones detector works by sliding detection window across image

• At each position, decide if desired object inside window

• Uses Haar features

Image source: Mercedes-Benz

Image source: Siemens

Basic algorithms & approaches!

!

3. Lane detection

• apply filter to image, ex Canny to find edges

• apply a Hough Transform to detect lines

• filter these lines input image

Images source: http://www.transistor.io/revisiting-lane-detection-using-opencv.html

Basic algorithms & approaches!

!

4. Obstacle detection & tracking

Monocular appearance based methods:

• use colour and shape to detect image regions that belong to obstacles

Monocular motion based methods:

use motion of image features and optical flow

• optical flow = pattern of apparent motion of image objects between two consecutive frames caused by the movement of object or camera (definition adapted from wiki)

• you have a set of points in an image, find the same points in another image

Image source: OpenCV

Stereo methods use the disparity map to get depth of image features. => detection of the ground plane (if it is unknown) and the detection of obstacles = any feature not on the ground plane

Basic algorithms & approaches!

!

4. Obstacle detection & tracking - continued

A lot of research & proposed solutions, some examples:

• Feature based detection (ex: using Haar features): Haar features =(sum of pixels in black region) - (sum of pixels in white region)

• Detection by background subtraction

• Many other methods, most based on stereo imaging

Image source: behance.net Image source: atlas.web.ua.pt Image source: http://reason.cs.uiuc.edu/

Other tips• only process what's needed/required from an image,

ex: processing road lanes, remove the sky/horizon from the image, you only want to process the road pixels (this is called Region Of Interest = ROI)

• use pre-defined, pre-computed values, for example you calibrate the camera once, and store the obtained results, re-use them when needed

Drive Assist App• monocular system based on Android devices

• fast, cheap & robust object detection (why? => use a mobile phone, portable, everyone has a phone, only requirement is to use a windshield mount for the phone

• not new, or breakthrough app, but performs better than competition

• real time processing & feedback (displaying the results)

Drive Assist AppSome testing results & effective range

Images source: http://www.driveassistapp.com

Drive Assist AppPerformance test

Drive Assist - Comparison

http://bit.ly/DriveAssistComp

Drive Assist - Demo

http://bit.ly/DriveAssistDemo

Q&A

Thank you!

Razvan Itu www.driveassistapp.com

www.mobileway.net twitter.com/razvan