KONGU ENGINEERING COLLEGE,

Perundurai, Erode.

Department of Mechatronics

ADVANCED FATIGUE DETECTION AND ACCIDENT PREVENTION

SYSTEM

1) A.V.Hari Raja,(II-Mechatronics),

E-mail: avhariraja@gmail.com

Mobile: 9150422617

2) Kingsley Anand Shanmugam,(II-Mechatronics),

E-mail: kingsleyanandshanmugam@rocketmail.com

Mobile: 918754123915

ABSTRACT

Recent days the traffic accident

has increased in great scale. Some of the

major factors are Driver drowsiness and

fatigue. The purpose of this paper is to

advance a system to detect fatigue

symptoms in drivers and produce timely

warnings that could prevent accidents.

This presents a real-time approach for

detection of driver’s fatigue. The

algorithm developed is unique to any

currently published papers, which was a

primary objective of this paper. Here the

eyes are located by a camera and the

intensity changes in the eye area

(including the effect of red pupil)

determine whether the eyes are open or

closed. If the eyes are found closed for 5

consecutive frames, the system draws the

conclusion that the driver is falling asleep

and issues a warning signal. The

advantage in this system is capable of

working under reasonable lighting

conditions also. The image acquisition

system acquires images with consistent

photometric property under different

climatic / ambient conditions using near-

infrared (NIR) illuminator and CCD

camera.

Using Technique on For detecting a

neighbor car index,

Homographic function

Homometric function

Radar sensing property

Incase if the driver is still in

unconscious the system provides the

automatic braking to avoid the crash

over the adjacent cars. The system

analysis, the speed and distance of the

neighboring cars and respond such that

to prevent accidents, while applying

braking. Also during emergency cases

like heart attack, unconsciousness etc, it

will be more advantageous.

INTRODUCTION:

Driver fatigue resulting from sleep

deprivation or sleep disorders is an

important factor in the increasing number

of accidents on today’s roads. The main

purpose was to advance a system to detect

fatigue symptoms in drivers and produce

timely warnings that could prevent

accidents. In the trucking industry, 57%

fatal truck accidents are due to driver’s

fatigue. Driver fatigue is a significant

factor in a large number of vehicle

accidents. Recent statistics estimate that

annually 1,200 deaths and 76,000 injuries

can be attributed to fatigue related crashes.

The main components of the system

consists of a remotely located video CCD

camera, a specially designed hardware

system for real-time image acquisition and

for controlling the illuminator and the

alarm system, and various computer vision

algorithms for simultaneously, real-time

and non-intrusively monitoring various

visual bio-behaviors that typically

characterize a driver’s level of vigilance.

FUNCTIONAL DESCRIPTION:

The input to the system are images from

a video camera mounted in front of the

car, which then analyzes each frame to

detect the face region. The face is

detected by searching for skin color-like

pixels in the image. Then a blob

separation performed on the grayscale

image helps obtain just the face region.

In the eye-tracking phase, the face region

obtained from the previous stage is

searched for localizing the eyes using a

pattern-matching method. Templates,

obtained by subtracting two frames and

performing a blob analysis on the

difference grayscale image, are used for

localizing the driver’s eyes.

The eyes are then analyzed to detect if

they are open or closed. If the eyes

remain closed continuously for more

than a certain number of frames, the

system decides that the eyes are closed

and gives a fatigue alert. It also checks

continuously for tracking errors. After

detecting errors in tracking, the system

starts all over again from face detection.

The main focus is on the detection of

micro-sleep symptoms. This is achieved

by monitoring the eyes of the driver

throughout the entire video sequence.

The three phases involved in order to

achieve this are the following:

(i) Localization of the face,

(ii) Tracking of eyes in each frame,

(iii) Detection of failure of tracking.

Outline of the System

1) FACE DETECTION:

Normalized chromatic color

representations are defined as the

normalized r- and g components of the

RGB color space. This representation

removes the brightness information from

the RGB signal while preserving its color.

Further, the complexity of the RGB color

space is simplified by the dimensional

reduction to a simple RG color space. Skin

color models vary with the skin color of

the people, video cameras used and also

with the lighting conditions. Using the skin

color model the system filter out the

incoming video frames to allow only those

pixels with high likelihood of being skin

pixels. The system uses a threshold to

filter out the skin like pixels from the rest

of the image. The filtered image is then

binarized and blob operation performed to

detect the face region from the rest of the

image space. In order to reduce the

computational cost and speed up the

processing, each incoming frame is sub

sampled to a 160x120 frame.

2) TRACKING OF EYES:

The reference eye patterns for each user

are recovered previously by taking the

difference of two images. The eye blink

is used to estimate the position of the

eye. The eye templates are recovered by

taking a difference of the two images and

employing blob (area) operations to

isolate the eye regions. For the correct

detection of the eye templates, it is

required that there is no other motion of

the face other than the eye blinks. The

eye pattern consists of the eyes centered

at the center of the iris of the user. The

system searches for the open eyes

starting from the left eyes first and then

looks for the right eyes. If the scores for

the open eyes are reasonably higher than

the acceptance level and the system

decides that the eyes are open, it does not

search for the closed eye patterns in the

image.

3) DETECTION OF FAILURE:

The threshold scores fixed for the open

eyes consist of a minimum above which

the system decides that the eyes are

probably open. When the scores are

above the maximum threshold, the

system decides for sure that the eyes are

open and does not search for the closed

eyes in the image. But if the scores are

between the minimum and the maximum

limits, then the system searches the

image for closed eyes too in order to

remove any mismatches.

In case the eyes of the subject remain

closed for unusually long periods of

time, the system gives a fatigue alert.

The fatigue alert persists as long as the

person does not open his eyes. In case all

the matches fail, the system decides that

there is a tracking failure and switches

back to the face localization stage. As the

face of the driver does not move a lot

between frames, we can use the same

region for searching the eyes in the next

frame.

DISADVANTAGES:

(i) There were mismatches especially in

the case of closed eyes as the system

finds any part of the skin region as the

eye. Thus, there were misses due to

incorrect matching with the facial hair

for open eyes and other parts of the face

for the closed eyes.

(ii)The system could not track the eyes

when the subject wears glasses while

driving.

(iii) Also the system could not track the

eyes, when the subject’s head rotation is

above 45 degrees and head tilt up is above.

ADVANCED FATIGUE DETECTION

SYSTEM:

People in fatigue exhibit certain

visual behaviors easily observable from

changes in their facial features like eyes,

head and face. This system can

simultaneously and in real time monitor

several visual behaviors that typically

characterize a person’s level of alertness

while driving. These visual cues include

eyelid movement, pupil movement, and

face orientation. The fatigue parameters

computed from these visual cues are

subsequently combined to form a

composite fatigue index that can robustly,

accurately characterize one’s vigilance

level.

IMAGE ACQUISITION SYSTEM:

The purpose of image acquisition is to

acquire the video images of the driver face

in real time. The acquired images should

have relatively consistent photometric

property under different climatic/ ambient

conditions and should produce

distinguishable features that can facilitate

the subsequent image processing. To This

end, the person’s face is illuminated using

a near-infrared illuminator (NIR). The use

of infrared (IR) illuminator serves three

purposes: first, it minimizes the impact of

different ambient light conditions,

therefore ensuring image quality under

varying real-world conditions including

poor illumination, day, and night; second,

it allows producing the bright pupil effect,

which constitutes the foundation for

detection and tracking the proposed visual

cues. Third, since NIR is barely visible to

the driver, this will minimize any

interference with the driver’s driving.

Principle of Bright and Dark Pupil Effects

A bright pupil can be obtained if the eyes

are illuminated with a NIR illuminator

beaming light along the camera optical

axis at certain wavelength. At the NIR

wavelength, pupils reflect almost all IR

light they receive along the path back to

the camera, producing the bright pupil

effect, very much similar to the red eye

effect in photography. If illuminated off

the camera optical axis, the pupils appear

dark since the reflected light will not enter

the camera lens.

This produces the so-called dark

pupil effects. IR illuminator consists two

sets of IR LEDs, distributed evenly and

symmetrically along the circumference of

two coplanar concentric rings. The center

of both rings coincides with the camera

optical axis. The use of multiple IR LEDS

can generate a strong light such that the IR

illumination from the illuminator

dominates the IR radiation exposed to the

driver’s face, therefore greatly minimizing

the IR effect from other sources. This

ensures the bright pupil effect under

different climatic conditions.

The use of more than one LED also allows

to produce the bright pupil for subjects far

away (3 f) from camera. To further

minimize interference from light sources

beyond IR light and to maintain uniform

illumination under different climatic

conditions, a narrow band pass NIR filter

is attached to the front of the lens

Acquired Image with Desired Bright Pupil Effect and sharp pupil spot

PUPIL DETECTION AND TRACKING:

Pupil Detection And Tracking System Flowchart

ILLUMINATION INTERFERENCE

REMOVAL VIA IMAGE SUBTRACTION:

The detection algorithm starts with a pre-

processing to minimize interference from

illumination sources other than the IR

illuminator. This includes sunlight and

ambient light interference background

illumination interference removal

(a)The image field obtained with both

ambient and IR light.

(b)The odd image field obtained with only

ambient light.

(c)The image resulting from subtraction

of (b) from (a)

To uniquely detect pupils, other bright

areas in the image must be removed or

they may adversely affect pupil detection.

The background clusters removal is

accomplished by subtracting the image

with only ambient light illumination from

the one illuminated by both the IR

illuminator and the ambient light. The

resultant image contains the illumination

effect from only the IR illuminator,

therefore with bright pupils and relatively

dark background. This method has been

found very effective in improving the

robustness and accuracy of our eye

tracking

The system uses a video decoder that

detects from each interlaced image frame

(camera output)the even and odd field

signals, which is inner IR rings on to

produce the dark and bright pupil image

fields. Then the system separates each

frame into two image fields (even and

odd), representing the bright and dark

pupil then used to alternately turn the outer

and images separately. The even image

field is then digitally subtracted from the

odd image field to produce the difference image.

Block Diagram of Image Subtraction Circuitry

For determining the initial pupil position,

given the image resulted from the removal

of the external illumination disturbance,

pupils may be detected by searching the

entire image to locate two bright regions

that satisfy certain size, shape, and

distance constraints. To do so, a search

window scans through the image. At each

location, the portion of the image covered

by the window is examined to determine

its intensity distribution. The blob is then

validated based on its shape, size, its

distance to the other detected pupil, and its

motion characteristics to ensure it is a

pupil.

COMPUTATION OF EYELID

MOVEMENT PARAMETERS:

Eyelid movement is one of the visual

behaviors that reflect a person’s level of

fatigue. There are several ocular measures

to characterize eyelid movement such as

eye blink frequency, eye closure duration,

eye closure speed, and the recently

developed parameter PERCLOS.

PERCLOS measures percentage of eye

closure over time, excluding the time spent

on normal closure.

Another ocular parameter that could

potentially be a good indicator of fatigue is

eye closure/opening speed, i.e. the amount

of time needed to fully close the eyes and

to fully open the eyes. An eye closure

occurs when the size of detected pupil

shrinks to a fraction (say 20%) of its

nominal size. As shown in Figure below,

an individual eye closure duration is

defined as the time difference between two

consecutive time instants, t2 and t3,

between which the pupil size is 20% or

less of the maximum pupil size. And an

individual eye closure speed is defined as

the time period of t1 to t2 or t3 to t4,

during which pupil size is between 20%

and 80% of nominal pupil size,

respectively.

Definition of Eye Closure Duration and Eye Open/Close Speed

FACE ORIENTATION DETERMINATION:

The system recovers 3D face pose from a

monocular view of the face with full

perspective projection. There is a direct

correlation between 3D face pose and

properties of pupils such as pupil’s size,

inter-pupil distance, and pupils shape. The

followings are apparent from images

above.

(i)The inter-pupil distance decreases as the

face rotates away from the frontal

orientation.

(ii) The ratio between the average intensity

of two pupils either increases to over one

or decreases to less than one as face rotates

away or rotates up/down.

(iii)The shapes of two pupils become more

elliptical as the face rotates away or rotates

up/down.

(iv) The sizes of the pupils also decrease

as the face rotates away or rotates

up/down.

WHAT’S AFTER DRIVER’S

FATIGUE DETECTION?

The accidents can take place in a

fraction of second. For example when

person becomes drowsy, he loses his

control from driving. In this case we have

implemented the method to prevent this.

The main aim of this method is to

to wake the host from his

drowsiness.

to prevent the vehicle from

accident.

ALERTING DRIVER:

The first step in this method is to

provide a vibrating seat for the driver. The

vibration of the seat is done with the help

of the stepper motor to wake the driver

from his drowsiness.

ACCIDENT PREVENTION:

Even though the driver regains his

drowsiness, he might have lost his driving

control. So it is necessary to provide the

automatic braking.

The surrounding vehicle must be

analyzed for automatic braking, which is

one bys CCD(Charge Control Device)

camera and RADAR sensors. They are

placed at specific positions for identifying

the neighboring cars. The main aim are the

verification of obstacles and the detection

of obstacle boundaries. This allows to

analyze the situation for carrying out

emergency braking. The verification of

obstacles is done by analyzing the scaling

of obstacles as they get closer to the

camera. The CCD Camera detects the

obstacles within the 8 degree span. But in

times of snowfall it is inefficient. So to

overcome this CCD camera uses the

Homomorphic function. During weather

variation condition, camera works based

on Homomorphic function, which is to

develop frequency domain procedure i.e.

the appearance of an image by simultances

Gray Level Range using Butterworth high

pass filter in output of camera for better

reliable pictures. This helps in the

amplification of the low intensity

waveforms from the obstacles.

To start with a simple example,

consider the two trucks in Fig as a radar

hypothesis. Our goal is to verify or discard

this hypothesis by means of computer

vision. The feature we are going to use is

image scale. As the vehicle approaches the

obstacles, the image of the obstacles taken

by a forward-looking camera will grow in

size. This principle is well known for

humans as it is simply based on the

perspective transformation of our eye or –

in computer vision context - of the camera

lens. The principle of distance estimation

by relative scale in camera sequences is

well illustrated by the theorem on

intersecting lines.

The quantity that relates scale to

distance is the covered distance of the

ego-vehicle. If the car travels half the

distance to any obstacle, the size of the

imaged obstacle will double. On the other

hand, if the scale and traveled distance are

known, obstacle distance can be

computed. Surely we don’t want to wait

for the scale factor do double to estimate

distances. But as scale can be efficiently

computed in images, small scale changes

already allow for distance estimates. In

this paper we measure scale changes by

automatic tracking of template regions.

CCD camera image-detects the movable obstacles which is marked in red color.

RADAR reflected image represented white templates on the moving obstacles

A problem arises if the scale of

such a template region does not originate

from obstacles and therefore leads to false

results. Truly, such problem can arise if

no obstacle is contained in the image and

for instance figures painted on the street

are tracked. This leads to the question of

how one can distinguish such template

regions on the street from others on

obstacles. Both, a distant obstacle and the

street, are a plane (planar surface) in first

approximation. Under perspective

transformations planes undergo

homographic transformations (eight

degrees of freedom). Homographies

contain the normal of mapped planes and

therefore the homography of the street is

different to that of obstacles. Because the

visible surface of an obstacle is

approximately parallel to the camera

plane, its transformation can be modeled

by translation and scale (similarity

transformation). The key to distinguish

template regions on the street from others

on obstacles lies in checking if the

transformation is described by a similarity

transformation or by the homography

generated by the street.

INDEX DETERMINATION IN

NEIGHBOR VEHICLES:

The obstacle distance is measured

with the help of CCD camera and radar

sensors. Simultaneously both capture

images and CCD camera image is

homographic image which easily detects

distortion on comparison with those two

images but it leaves the movement of

leaves because it is in slight variation.

Variation distance determined by checking

two consecutive images, for checking

obstacle’s position. The radar sensors

recognize the obstacles observed in

homography image represented by dot

sequences.

CCD camera detects the obstacles

upto 100m. According to the frame

number, distortion due to distance

variation is varied which is mentioned in

the graph seen below. Actually CCD

camera and radar placed at front and back

of the vehicle.

Any vehicle on the side of the car

is determined by optical sensor in

temperature variation from neighbor

vehicle engine.

.

ABS BRAKING SYSTEM FOR

STOPPING VEHICLE:

ABS braking applied related to the

index of neighbor vehicles from the wheel

speed sensors. Apart from the wheel speed

sensors abs braking system consists of

electronic unit and hydraulic unit. The

electronic control unit monitors and

compares the signals from the wheel-speed

sensors. If the electronic control unit

senses rapid deceleration (impending lock-

up) at a given wheel, the electronic control

unit commands the hydraulic control unit

to reduce hydraulic pressure to that wheel.

Further for the indication of braking, side

lights are activated with the horn to alert

the neighboring vehicles.

CONCLUSION:

This fatigue monitor system was

tested in a simulating environment with

subjects of different ethnic backgrounds,

different genders, ages, and under different

illumination conditions. The system was

found very robust, reliable and accurate.

The following conclusions were made:

• Image processing achieves highly

accurate and reliable detection of

drowsiness.

• Image processing offers a non-invasive

approach to detecting drowsiness without

the annoyance and interference.

• Further the system provides driver

awakening alarm and the accident

prevention.

REFERENCES:

1) Gee, A. & Cipoll, R. (1994)

Determining the gaze of faces

images. Image and Vision

Computing,

2) “Remote sensing and image

interpretation” by Thomas

M.Lillesand and Ralph

W.Kiefor.

3) Wierville, W.W. (1994)

Overview of research on driver

drowsiness definition and

driver drowsiness

detection.ESV, Munich.