Report Final

55
A Project Report On College Attendance System based on Facial Recognition Submitted to Amity University, Uttar Pradesh In partial fulfilment of the requirements for the award of the degree of Bachelor of Technology In Computer Science & Engineering By Rohan Khurana Under the guidance of Ms. Divya Sharma Department of Computer Science & Engineering Amity School of Engineering & Technology Amity University, Noida Uttar Pradesh April, 2012

Transcript of Report Final

Page 1: Report Final

A Project Report

On

College Attendance System based on Facial Recognition

Submitted to

Amity University, Uttar Pradesh

In partial fulfilment of the requirements for the award of the degree of

Bachelor of Technology

In

Computer Science & Engineering

By

Rohan Khurana

Under the guidance of

Ms. Divya Sharma

Department of Computer Science & Engineering

Amity School of Engineering & Technology

Amity University, Noida

Uttar Pradesh

April, 2012

Page 2: Report Final

ii

DECLARATION

I, Rohan Khurana, student of Bachelor of Technology (Computer Science &

Engineering) hereby declare that the project titled “College Attendance System based

on Facial Recognition” which is submitted by me to Department of Computer Science

& Engineering, Amity School of Engineering and Technology, Amity University

Uttar Pradesh, Noida, in partial fulfilment of the requirement for the award of the

degree of Bachelor of Technology in Computer Science & Engineering, has not been

previously formed the basis for the award of any degree, diploma or other similar title

or recognition.

Place: Noida

Date: 30th April 2012 Rohan Khurana

Page 3: Report Final

iii

CERTIFICATE

On the basis of declaration submitted by Rohan Khurana, student of B. Tech

(Computer Science & Engineering), I hereby certify that the project titled “College

Attendance System based on Facial Recognition” which is submitted to Department

of Computer Science & Engineering, Amity School of Engineering and Technology,

Amity University Uttar Pradesh, Noida, in partial fulfilment of the requirement for the

award of the degree of Bachelor of Technology in Computer Science & Engineering,

is an original contribution with existing knowledge and faithful record of work carried

out by him/them under my guidance and supervision.

To the best of my knowledge this work has not been submitted in part or full for any

Degree or Diploma to this University or elsewhere.

Noida (Ms. Divya Sharma)

Date Department of Computer Science and Engineering

Amity School of Engineering and Technology

Amity University Uttar Pradesh, Noida

Page 4: Report Final

iv

ACKNOWLEDGEMENT

“College Attendance System based on Facial Recognition” is a biometrics oriented

project aimed at presenting some concepts from various computer science disciplines

with a programming approach. An effort has been made to present this material in a

very lucid manner followed by scrutinized observation to every step of the source

code so implemented.

I take this opportunity to express my deep regards to respected Ms. Divya Sharma,

Dept. of Computer Science & Engineering, A.S.E.T, AUUP. I am indebted to her for

providing me the necessary counselling and support. I am deeply indebted to our

mentor Miss. Divya Sharma for encroachment upon her time. I would like to express

a heartfelt gratitude for her insight, expertise and energy contributed greatly to this

project. In addition to this I acknowledge my debt of gratitude to the numerous

authors whose great masterly work I have consulted during the preparation of this

project.

Last but not the least; I would like to thank my parents for their sound counselling and

cheerful support.

I hope that this project on “College Attendance System based on Facial Recognition”

in the present form caters to the needs of all concerned.

Rohan Khurana

Page 5: Report Final

v

ABSTRACT

This research aims at providing a system to automatically record the students’

attendance during lecture hours in a hall or room using facial recognition technology

instead of the traditional manual methods.

The objective behind this research is to thoroughly study the field if pattern

recognition (facial recognition) which is very important and is used in various

applications like identification and detection.

Page 6: Report Final

vi

CONTENTS

Declaration iii

Certificate iv

Acknowledgements v

Abstract vi

Contents vii

CHAPTER 1 Introduction 1

1.1 Background of the Study 1

1.2 Problem Definition 1

1.3 Proposed Scope and Enhancement 2

CHAPTER 2 Literature Review and Prior Research 3

2.1 Biometric Recognition 3

2.1.1 Emergence 3

2.1.2 Advantages 4

2.1.3 Types 5

2.1.4 Modules of a Biometric System 8

2.2 Facial Recognition 10

2.2.1 Automatic Facial Recognition Process Flow 10

2.2.2 Facial Recognition Techniques 12

2.2.3 Limitations of Face Recognition Technologies 14

CHAPTER 3 Materials and Methodology 18

3.1 System Development Requirements 18

3.1.1 Hardware 18

3.1.2 Software 18

3.1.3 Internet Connection 18

3.2 System Analysis and Design 19

3.2.1 Logical Data Flow (DFD) 19

3.2.2 Data Structures 23

3.2.3 Database Design(ERD) 24

Page 7: Report Final

vii

CHAPTER 4 Implementation 25

4.1 Introduction 25

4.2 Colour Based Techniques 25

4.2.1 Description 25

4.2.2 Problems Faced 26

4.2.3 Limitations 26

4.2.4 Recommendations 26

4.2.5 Flow Chart 27

4.3 Principal Component Analysis 28

4.3.1 Description 28

4.3.2 PCA based Face Recognition Algorithm 29

4.3.3 Calculation of Eigenfaces with PCA 35

4.3.4 Classifying Images 39

4.3.5 Input and Output 40

4.3.6 MATLAB Implementation 41

4.3.7 Database Implementation 41

CHAPTER 5 Discussion of Result 42

CHAPTER 6 Presentation of Result 43

6.1 The Basic User Interface 43

6.2 Average Face Output 44

6.3 Recognition Output 45

CHAPTER 7 Conclusions 46

CHAPTER 8 Future Prospects 47

CHAPTER 9 References 48

Page 8: Report Final

1

1. Introduction

1.1 Background of the study

The rapid development in the field of pattern recognition and its uses in different

areas e.g. (signature recognition, facial recognition), arises the importance of the

utilization of this technology in different areas in large organizations. This is mainly

because these applications help the top-management take decisions that improve the

performance and effectiveness of the organization. On the other hand, for an

organization to be effective, it needs accurate and fast means of recording the

performance of the people inside this organization. Biometric recognition has the

potential to become an irreplaceable part of many identification systems used for

evaluating the performance of those people working within the organization.

Although biometric technologies are being applied in many fields it has not yet

delivered its promise of guaranteeing automatic human recognition. This research is

the first of its kind to attempt to provide an automated attendance system that

recognizes students using face recognition technology through an image/video stream

to record their attendance in lectures or sections and evaluating their performance

accordingly.

1.2 Problem definition

Every time a lecture or laboratory starts, the lecturer delays the lecture to record

students’ attendance. This is a lengthy process and takes a lot of time and effort,

especially if it is a lecture with a huge number of students. It also causes a lot of

disturbance and interruption when an exam is held. Moreover the attendance sheet is

subjected to damage and loss while being passed on between different students or

teaching staff. And when the number of students enrolled in a certain course is huge,

the lecturers tend to call a couple of student names at random which is not a fair

student evaluation process either. Finally, these attendance records are used by the

staff to monitor the students’ attendance rates. This process could be easy and

effective with a small number of students but on the other hand, dealing with the

records of a large number of students often leads to human errors.

Page 9: Report Final

2

1.3 Proposed scope and enhancement

This project proposes solutions to all the above mentioned problems by providing an

automated attendance system for all the students that attend a certain lecture,

laboratory or exam at its specific time, thus saving time, effort and reducing

distractions and disturbance. Another advantage concerning exams, is when the

invigilator or the advisor accidentally loses an exam paper or the student lies about

attending the exam, there will be a record of the students’ attendance for the exam at

that time, thus protecting both lecturer’s and student’s rights. In addition, an

automated performance evaluation would provide more accurate and reliable results

avoiding human error.

Page 10: Report Final

3

2. Literature review and prior research

I have studied the literature available in the fields of pattern recognition and biometric

systems with a focus on facial recognition. In addition, a study of previous attendance

systems and its atomization attempts was conducted to examine the similar previous

systems. The literature is divided into two main parts; the first part examines the

different biometric characteristics and systems. In the second part, an in-depth study

on facial recognition is conducted for it is the technology to be used by our proposed

project.

2.1 Biometric recognition:

There are three different types of authentication: something you know, such as

passwords, something you have, such as badges or cards, and finally something you

are, which mainly depends on biometrics and physical traits. Each of these three

authentication methods has its advantages and disadvantages, and each is considered

appropriate for certain types of application.

Among there three types, scientists and researchers consider biometric recognition

systems as high-level security systems. They define the biometric term as the

“science, involving the statistical analysis of biological characteristics”. Biometric

recognition generally matches a live digital image of a portion from a certain physical

body part with a previously recorded image of that same portion; whether it is in

identification mode where one-to-many comparisons take place, or verification

(authentication) mode where one-to-one comparisons occur.

2.1.1 Emergence

For thousands of years, humans have used biometrics and physical characteristics

such as face and voice to identify other human beings for a wide variety of purposes,

ranging from simple business interactions to criminal investigations. The study of

fingerprint dates back to ancient China. While the ancient Jordanian have commonly

used physical traits such as height, scars, eye and hair colour to identify individuals

for business transactions. Generally speaking, the human mind automatically

recognizes different people depending on their external physical and behavioural

features.

Page 11: Report Final

4

2.1.2 Advantages

The main advantage of biometric systems over normal automated systems is that they

really do what they are supposed to do, which is authenticating the user, in a way

imitating the human capabilities; and using real human physical characteristics, which

are almost impossible to change. In addition, some researchers proposed that

biometrics are not subjected to theft, loss or passing to anyone else like what is done

with cards or passwords. While some others object and point out that they are not a

secret and could be in many cases falsified or stolen from computer systems.

Another advantage of using biometrics would be speed in doing work. For example,

authentication of a regular user with iris recognition would take two to three seconds,

while using a key to open the door would need five to ten seconds. Moreover the

normal human perception system has an impressive capability of recognizing and

distinguishing different faces, but this capability is limited by the number and type of

faces that can be easily processed. This limitation is overcome by the computerized

facial recognition systems that can mimic the human mind capability in addition to

storing and processing as much people as necessary. And finally, it is important to

point out that those technologies that are able to support biometric based applications

are becoming more and more available, which make biometric based systems more

available to users.

Page 12: Report Final

5

2.1.3 Types

Any human body characteristic that is considered universal, distinctive, almost

permanent, collectable and clear enough not to fool the capturing system, can be used

as a biometric feature if it is accepted to be used by humans. Besides that, biometric

systems now come in all different shapes and sizes. It ranges from hardware,

software, OEMs, certain modules or complete solutions. In this section of our

literature review, we will quickly go through the different biometric types used for

person detection.

2.1.3.1 The Eye

Analysing the eye is generally thought to present the highest levels of accuracy and

uniqueness. They can be divided into two different technologies: iris biometrics and

retina biometrics.

Iris: It is the coloured tissue representing the ring surrounding the eye pupil.

Each person’s iris has a unique structure and a complex pattern. In addition, it

is believed that artificially duplicating an iris is virtually impossible. It is also

known that the iris is from the first body parts decaying after death, therefore

it is unlikely that a dead iris could be used to by-pass a biometric system.

Retina: It is the layer of blood vessels situated at the back of the eye. Just as

the iris, the retina forms a unique pattern and decays quickly after death.

Retina recognition systems are complex but at the same time regarded as the

most secure biometric system.

2.1.3.2 Face

For a computerized system to mimic the human ability in recognizing faces,

sophisticated and complex artificial intelligence and machine learning is needed to be

able to compare and match human faces with different poses, facial hair, glasses, etc.

That is why these systems depend greatly on the extraction and comparison engines.

Different tools maybe used in these systems such as standard video, or still imaging or

thermal imaging.

Page 13: Report Final

6

2.1.3.3 Finger

In this type of study, the finger tips are analysed for its unique pattern and print

produced by the finger minutiae, in addition to finger geometry which concentrates on

the shape of the finger rather than the print itself.

2.1.3.4 Hand

There are mainly three biometric systems using the characteristics of the human hand.

These are: Hand geometry, Vein pattern analysis, and Palm identification.

Hand geometry: Just as with the finger geometry, the hand geometry

analysing system captures a three-dimensional image of the hand and

measures the shape and length of fingers and knuckles. Hand geometry was

from those first biometric systems used for access control applications.

Although it is not very accurate, it is thought to be very convenient and large

amounts can be processed quickly.

Vein pattern analysis: The past number of years has witnessed much

development in the area of analysing the patterns of the veins in the back of

the human hand. This technique is called the “vein check” which examines

the unique pattern of the blood vessels or what is called the “vein tree” which

can be captured by the infra-red light.

Palm identification: Palm biometrics is very similar to the study of finger

print; where the print and the patterns of ridges, valleys and minutiae found

on the palm are analysed and studied.

2.1.3.5 Voice

Voice biometrics focus on the sound of the voice, not on what is being said. That is

why it is important to distinguish between this technology and those technologies

recognizing words and commands. The sound of the human voice is caused by

vibration in the vocal chords. The shape and size of the vocal tract, in addition, the

shape of the mouth and the nasal cavities all contribute in the way a voice sounds.

Voice recognition techniques may either use text-dependent or text-independent

methods. In other words, voice may be captured by speaking out any phrase, word or

number (text-independent) or by specifically saying a certain password combining

phrases, words or numbers (text-dependent). However this biometric system may be

Page 14: Report Final

7

challenged by the background noise that reduces the quality of the data and the

system’s performance.

Page 15: Report Final

8

2.1.4 Modules of a Biometric System

Biometric systems are designed using the following four modules as explained in

figure 1:

Figure 1: Typical biometric recognition system

2.1.4.1 Sensor or Capturing Module

This module captures the biometric data. Different devices are used for different

biometric system according to the feature that is being captured; for example, video

camera, thermal camera, voice recorder, infrared… etc.

2.1.4.2 Feature Extraction Module

This module processes the captured biometric data to extract a set of discriminatory

features forming a template. For example: the position and orientation of minutiae

points in a fingerprint image are extracted in the feature extraction module of a

fingerprint-based biometric system.

2.1.4.3 Matcher Module

In this module, the features extracted from the previous module are compared to those

stored and decisions regarding matching score are made. For example: in a finger-

print based biometric system, the number of matching minutiae points determine the

matching score and consequently the match or not results.

Page 16: Report Final

9

2.1.4.4 System Database Module

It is used by the biometric system to store the templates of the enrolled users. The

enrolment module is responsible for enrolling individuals into the biometric system

database, where the biometric characteristic of an individual is first scanned by the

sensor to produce a digital representation (initial stored template) of the characteristic.

Page 17: Report Final

10

2.2 Facial Recognition

Face recognition is considered to be one of the most successful applications of image

analysis and processing. That is the main reason behind the great attention it has been

given in the past several years.

2.2.1 Automatic Facial Recognition Process Flow

Generally any biometric system goes through the same processes of the four modules

explained earlier, biometric capture, feature extraction, and comparison with

templates available in the database as explained in figure 1. The facial recognition

process is similar to the general biometric recognition process. In the face-base

biometric systems detection, alignment, feature extraction, and matching take place.

Figure 2.5.2 Face recognition processing flow

The facial recognition process can be divided into two main stages: processing before

detection where face detection and alignment take place (localization and

normalization), and afterwards recognition occur through feature extraction and

matching steps.

Face Detection: This process separates the facial area from the rest of the

background image. In the case of video streams, faces can be tracked using a

face tracking component.

Face Alignment: This process focus on finding the best localization and

normalization of the face; where the detection step roughly estimates the

position of the face, this step outlines the facial components, such as face

outline, eyes, nose, ears and mouth. Afterwards normalization with respect to

geometrical transforms such as size and pose, in addition to photometrical

properties such as illumination and grey scale take place.

Page 18: Report Final

11

Feature Extraction: After the previous two steps, feature extraction is

performed resulting in effective information that is useful for distinguishing

between faces of different persons and stable with respect to the geometrical

and photometrical variations.

Face Matching: The extracted features are compared to those stored in the

database, and decisions are made according to the sufficient confidence in the

match score.

Page 19: Report Final

12

2.2.2. Facial Recognition Techniques

This section gives an overview on the major human face recognition techniques that

apply mostly to frontal faces.

2.2.2.1 Eigen faces

Eigen face is one of the most thoroughly investigated approaches to face recognition.

This method would be less sensitive to appearance changes than the standard Eigen

face method.

2.2.2.2 Neural Networks

The attractiveness of using neural networks could be due to its non-linearity in the

network. In general, neural network approaches encounter problems when the number

of classes (i.e., individuals) increases.

2.2.2.3 Graph Matching

Graph matching is another approach to face recognition presented a dynamic link

structure. In general, dynamic link architecture is superior to other face recognition

techniques in terms of rotation invariance; however, the matching process is

computationally expensive.

2.2.2.4 Hidden Markov Models (HMMs)

Stochastic modelling of non-stationary vector time series based on (HMM) has been

very successful for speech applications applied this method to human face

recognition. Faces were intuitively divided into regions such as the eyes, nose,

mouth, etc., which can be associated with the states of a hidden Markov model.

2.2.2.5 Geometrical Feature Matching

Geometrical feature matching techniques are based on the computation of a set of

geometrical features from the picture of a face. Current automated face feature

location algorithms do not provide a high degree of accuracy and require considerable

computational time.

Page 20: Report Final

13

2.2.2.6 Template Matching

A simple version of template matching is that a test image represented as a two-

dimensional array of intensity values is compared using a suitable metric, such as the

Euclidean distance, with a single template representing the whole face. In general,

template-based approaches compared to feature matching are a more logical

approach.

2.2.2.7 3D Morphable Model

The morphable face model is based on a vector space representation of faces that is

constructed such that any convex combination of shape and texture vectors of a set of

examples describes a realistic human face.

Page 21: Report Final

14

2.2.3 Limitations and Challenges of Face Recognition Technologies

As mentioned earlier, face recognition technology, just as any other biometric

technology, has not yet delivered it promise. In spite of all its potentials, it is still

quite limited in its applied scope. Many researchers have identified different problems

for the biometric system; they can be categorized in four main challenges:

2.2.3.1 Accuracy

Two biometric samples collected from the same person are not exactly the same due

to the imperfect imaging conditions. In addition, the face recognition technology is

not robust enough to handle uncontrolled and unconstrained environments. In

consequence, the results accuracy is not acceptable.

As explained in figure 3, inaccuracy can occur in two different forms, either False

Non-Match (false reject / Type 1 error) in which the system falsely declares the

failure of match between the instance and the correct stored template; or False Match

(false accept / Type 2 error) in which the system incorrectly declares a successful

match between the instance and one of the templates in the database.

Figure 3: Biometrics System Errors

Page 22: Report Final

15

These errors are mainly caused by the complexity and difficulties of the recognition

process because of the uncontrollable variables such as lighting, pose, expression,

aging, weight gain or loss, hairstyle and accessories; figures 4, 5 and 6 present

examples for some types of variations. This challenge is reduced as more training data

is available to enhance the performance of the biometric system.

Figure 4: Position, lighting, expression

Figure 5: Occlusion/blockage/hiding of some features

Figure 6: Ageing

Page 23: Report Final

16

2.2.3.2. Scale

The number of entities enrolled in the database of the biometric system greatly affects

the speed, accuracy and performance in case of identification systems; where each

new instance is compared to all those in the database to find a match. Unlike the

verification where one to one comparison takes place. As mentioned in this article,

according to some studies conducted, a facial authentication takes about 90 msec,

while a large scale ID throughput needs about 0.66 min. That is why, in biometric

identification systems, it is absolutely necessary to find more efficient ways for

biometric comparison. This can be reached by performing course pattern

classification. For example: the captured face is firstly classified into male or female,

then, an age range can be determined… etc. In this way, the search in the system’s

database can be narrowed down in an efficient way. The large intra-class variation

still represents a great challenge in the field of pattern recognition, where many

studies are being conducted to find the best way for indexing patterns similar to the

ways used in conventional databases.

2.2.3.3. Security

Facial recognition and other biometric systems are used for many security

applications, claiming that biometrics is a secure way of authenticating access. But in

fact, security of biometrics (especially face), is very questionable. This is caused by

two main reasons:

Biometrics is not a secret: This means that anyone including the attacker

knows exactly the biometric features of the targeted user.

Biometrics is not recoverable: This means that one cannot change his face

in case it became compromised.

2.2.3.4 Privacy

The issue of using recognition based systems has raised many concerns of possible

privacy violations; which is a major concern in many locations, such as the American

Civil Liberties Union (ACLU) which opposes the use of face recognition software at

airports due to privacy concern. The database of a biometric system hold irrefutable

proof of one’s identity; and there are no regulations or guarantees on how these

Page 24: Report Final

17

information might be used or what it could be used for. These privacy issues mostly

result in the reluctance of users to use these biometric systems.

On the other hand some people propose that biometrics is a privacy protection tool

rather than intrusion to civil rights. This would be achieved through managing data

protection and encryption along the biometric system.

Page 25: Report Final

18

3 Materials and Methodology

This chapter enlists the system requirements for the development of this project as

well as the system designs implemented for the project.

3.1 System Development Requirements

3.1.1 Hardware

1) A camera

2) A desktop or laptop computer with the following specifications:

Intel core 2 duo processor with 2.4 MHz or higher clock rate and 2 MB or

higher cache

3GB RAM or higher

250GB storage space

3.1.2 Software

1) Operating system (Windows 7)

2) MATLAB

3) Database (MySQL)

4) Video for Windows (VFW)

5) Camera Driver

3.1.3 Internet Connection

1) Any way of connecting to the internet (For confirmation emails)

Page 26: Report Final

19

3.2 System Analysis and Design

3.2.1 Logical Data Flow (DFD)

The data flow diagram is a diagram that shows how the data manipulated by a system

flows through the various processes. It provides no information about the timing or

ordering of processes, or about whether processes will operate in sequence or in

parallel. It is therefore quite different from a flowchart, which shows the flow of

control through an algorithm, allowing a reader to determine what operations will be

performed, in what order, and under what circumstances, but not what kinds of data

will be input to and output from the system, nor where the data will come from and go

to, nor where the data will be stored (all of which are shown on a DFD). DFD consists

of four main symbols which are:-

External Entities

Data stores

Processes

Dataflow lines

The system will be illustrated on three different levels of DFD which are Context

diagram (level 0)( Figure 7) which consists of all the external entities, one main

process only and the data flow between them. It has no further details. Then is drawn

the next level of DFD which is High level diagram (level 1) ( Figure 8) in which the

data stores are shown and the main single process is broken down into the major high-

level, processes that are done by the system and the dataflow between them. The third

level of DFD, Low diagram (level 2) ( Figures 9 and 10) has a purpose of breaking

down the small, main processes into even smaller processes (children processes) to

simplify their work to become clearer to the reader. Together they wholly and

completely describe the parent process, and combined must perform its full capacity.

This decomposition of the parent process is called explosion of the process.

System with proposed automated attendance module: In the figures below is the

proposed system data flow that will be operating on automated facial recognition

attendance recording methods.

Page 27: Report Final

20

Context Diagram (Level 0):

Figure 7: (Context Diagram- Level 0)

High Diagram (Level 1):

Figure 8: (High Diagram- level 1)

Page 28: Report Final

21

Low level Diagram (Level 2): This level is the level where we explode or break

down the process into smaller ones. We will only examine the segmentation fig and

the face detection fig 3.4 processes because these are the processes where the

automation and most of the project work will be take place.

Figure 9: (Low Level Diagram- Level 2 (Process 2))

Page 29: Report Final

22

Figure 10: (Low Level Diagram- Level 2 (Process 3))

Page 30: Report Final

23

3.2.2 Data Structure

Course DS = Course ID + Course Name

Student DS = Student ID + First Name + Last Name + Email

Attendance = Attendance ID + Attendance Date

Attendance Date = Day + Month + Year

Page 31: Report Final

24

3.2.3 Database Design (ERD)

A diagram that pictorially represents entities, the relationships between them and the

attributes used to describe them in a database. The figure below illustrates the

attributes of each entity and the relationships between them. This figure shows the

types of the entities and their lengths, data types and relations between the tables.

Figure 11: (ERD)

Page 32: Report Final

25

4 Implementation

4.1 Introduction

In this project different techniques have being studied like colour based detection and

Principle Component Analysis (PCA) for face detection and for feature extraction,

PCA and Linear Discriminate Analysis (LDA). For detection, Colour based technique

was implemented, which depends on the detection of the human skin colour with all

its different variations in the image. The skin area of the image is then segmented and

passed to the recognition process. For recognition, PCA technique has been

implemented which a statistical approach that deals with pure mathematical matrixes

not image processing like the colour based technique used for detection. PCA can also

be used for detection.

4.2 Colour based technique

4.2.1 Description

In this technique the image is converted into a new colour space, HCbCr where H is

the Hue component of the HIS colour space and the Cb and Cr components are

calculated by the formulas stated in Equation 4.1.

Cb= (0.148 * R) - (0.291*G) + (0.439 * B) + 128

Cr= (0.439 * R) - (0.368 * G) - (0.071 * B) + 128

Equation 4.1

The new image is scanned pixel by pixel and if each pixel satisfies the condition

stated in Equation 4.2, which is the range of the human skin, its value is set to white

else it is set to black.

Page 33: Report Final

26

If (H(x,y)>=.01) && (H(x,y)<=0.1 ) &&

(Cr(x,y)>=140) && (Cr(x,y)<=165) &&

(Cb(x,y)>=140) && (Cb(x,y)<=195)

Set pixel value to white

Else

Set pixel value to black

Equation 4.2

The resultant image is converted into black white colour space in which pixels have

values of 0 and 1 only (black and white). The black white image is sent to the

segmenting function which gets the x axis and y axis coordinates of the face area to

crop it out of the original image to remove the rest of the data in the image that is not

important like the background.

4.2.2 Problems faced

After applying this technique there was a problem that there was noise that

tremendously affected the segmentation process so a noise removal filter was

implemented to successfully eliminate this problem. But it reduces the face area a

little, which is acceptable.

4.2.3 Limitations

1. This algorithm should be applied on one face at a time. It doesn’t support the

detection of multiple faces.

2. It is tremulously affected by the environment’s lighting conditions.

4.2.4 Recommendations

1. To use this technique the lighting conditions should be well controlled.

2. A better technique to solve these problems is to use PCA in the detection of the

face, which also supports multiple face detection.

Page 34: Report Final

27

4.2.5 Flow chart

Figure 12: Colour based technique

Page 35: Report Final

28

4.3 Principal Component Analysis

4.3.1 Description

Based on an information theory approach, PCA decomposes face images into a small

set of characteristic features images called “Eigenfaces” that can be described as the

principal components of the initial training set of images. In the language of

information theory, the extractions of the most applicable information in a face image

is desired, and then encoded as effectively as possible. Then it compares one face

encoding with a database of models encoded similarly. In PCA, one can transform

each original image of the training set into a corresponding eigenface. Therefore, one

important feature of PCA is that the reconstruction of any original image from the

training set by combining the eigenfaces is possible. Eigenfaces are the characteristic

features of the faces. Therefore one could say that the original face image can be

reconstructed form eigenfaces if one adds up all the eigenfaces (features) in the right

proportion. Each eigenface represents only certain features if the face, which may or

may not be present in the original image.

If the feature is present in the original image to a higher degree, the share of the

corresponding eigenface in the sum of the eigenfaces should be greater. On the other

hand if the particular feature is not present in the original image, the corresponding

eigenface should contribute a smaller part to the sum of eigenfaces. Indeed, in order to

reconstruct the original image form the eigenfaces, one has to build a kind of

weighted sum of all the eigenfaces. That is, the reconstructed original image is equal

to a sum of all eigenfaces, with each eigenface having a certain weight. This weight

specifies, to what degree the specific features (eigenface) is present in the original

image. If all the eigenfaces extracted from the original images are used, one can

reconstruct the original images from the eigenfaces exactly. But using only a part of

the eigenfaces is applicable. Hence, the reconstructed image is an approximation of

the original image. However, losses due to omitting some of the eigenfaces can be

minimized, which is achieved by selecting only the most important features

(eigenfaces).

Page 36: Report Final

29

Moreover, it is possible not only to extract the face from eigenfaces given a set of

weights, but to the extract the weights from eigenfaces and the face to be recognized.

These weights act as the amount by which the face differs from the “typical” face

represented by the eigenfaces. Therefore, using these weights one can determine two

important things:

1. Check whether the image is a face. In the case the weights of the image differ too

much from the weights of face images, the image probably not a face.

2. Similar faces (images) possess similar features (eigenfaces) to similar degrees

(weights). If weights from all images available are extracted, the images could be

grouped to clusters. Thus, all images having similar weights are likely to be a similar

face.

4.3.2 PCA-based face recognition algorithm

The approach to PCA-based face recognition involved the following initialization

operations:

4.3.2.1 Initialization

Acquire an initial set of face images (the training set). Calculate the eigenfaces from

the training set, keeping only the M images that correspond to the highest

eigenvalues. These M images define the face space. As new faces are experienced, the

eigenvalues can be updated or recalculated. Calculate the corresponding distribution

in M-dimensional weight space for each known individual, by projecting their face

images onto the face space. Having initialized the system, the following steps are used

to recognize new faces: Calculate a set of weights based on the input image and the M

eigenfaces by projecting the input image onto each of the eigenfaces. Determine if the

image is a face at all (whether known or unknown) by checking to see if the image is

sufficiently close to the face space. If it is a face, classify the weight pattern as either a

known person or as unknown. The coming sections will elaborate the steps needed to

perform the PCA using eigenfaces on a set of images in detail.

Page 37: Report Final

30

4.3.2.2 Main Algorithm phases

The previous initialization processes can be summed up into three main phases. Three

main functional units are involved in these phases. The characteristics of these phases

are described below:

4.3.2.2.1 Face database formation phase

During this phase, the gathering and the pre-processing of the face images that are

going to be added to the face database are performed. Face images are stored in a face

library (file system) in the system. Every action such as training set or eigenface

formation is performed on this face library. In order to start the face recognition

process, the face library has to be filled with face images. Weight vectors of the face

library members are empty until a training set is chosen and eigenvectors are formed.

4.3.2.2.2 Training phase

Images that are going to be in the training set are chosen from the entire face library.

After choosing the training set, eigenfaces are formed and stored for later calculations.

Eigenfaces are calculated from the training set, keeping only the M images that

correspond to the highest eigenvalues. These M eigenfaces define the M-dimensional

face space. When the new faces are acknowledged, the eigenfaces can be updated or

recalculated. The corresponding weight vector of each face library member has now

been updated.

4.3.2.2.3 Recognition and learning phase

After choosing a training set and constructing the weight vectors of face library

members, now the system is ready to perform the recognition process. The

recognition process in initialized by choosing the input image (the image, one seeks to

recognize). The weight vector is constructed with the aid of the eigenfaces that were

already stored during the training phase. After obtaining the weight vector, it is

compared with the weight vector of every face library member with a user defined

“threshold”. If there exists at least one face library member that is similar to the

acquired image within that threshold then, the face image is classified as known”.

Otherwise, a miss has occurred and the face image with its corresponding weight

vector for later use. This process is called learning to recognize.

Page 38: Report Final

31

4.3.2.3 The Use of Eigenfaces for Recognition

4.3.2.3.1. Overview of the algorithm using eigenfaces

The algorithm for the facial recognition system using eigenfaces is basically described

as follows:

First, the original images of the training set are transformed into a set of eigenfaces E.

Afterwards, the weights are calculated for each image of the training set and stored in

the set W.

When observing an unknown image X, the weights are calculated for that particular

image and stored in the vector Wx.

Afterwards, Wx is compared with the weights of images, of which one knows for

certain that they are faces (the weights of the training set W).One way to do it would

be to regard each weight vector as a point in space and calculate an average distance

D between the weight vectors from Wx and the weight vector of the unknown image

Wx (the Euclidean distance described in the appendix B would be a measure for that).

If this average distance exceeds some threshold value, then the weight vector of the

unknown image Wx lays too far apart form the weights of the faces. In this case, the

unknown X is considered not a face. Otherwise (if X is actually a face), its weight

vector Wx is stored for later classification.

4.3.2.3.2 Eigenvectors and eigenvalues definitions

An eigenvector of a matrix is a vector such that, if multiplied with the matrix, the

result is always an integer multiple of that vector. Its direction is not changed by that

transformation. This integer value is the corresponding eigenvalue of the eigenvector.

The corresponding eigenvalue is the proportion by which an eigenvector’s magnitude

is changed. This relationship can be described by the equation M * u =v * u, where u

is an eigenvector of the matrix M and v is the corresponding eigenvalues. This means,

an eigenvalue of 2 states that the length of the eigenvector has been doubled. An

eigenvalue of 1 means that the length of the eigenvector stays the same.

Page 39: Report Final

32

Eigenvectors possess following properties:

They can be determined only for square matrices.

There are n eigenvectors (and corresponding eigenvalues) in an n * n matrix.

All the eigenvectors are perpendicular, i.e. at right angle with each other.

4.3.2.3.3 Problems faced

Calculating the average and weights of the faces takes so much processing and time

when run for the first time to run the program.

4.3.2.3.4 Limitations

Non-uniform backgrounds and lighting conditions affect the recognition process.

4.3.2.3.5 Recommendation

Using black or uniform background used in the image.

Use sufficient light to illuminate the scene

Page 40: Report Final

33

4.3.2.3.6 Flow chart

Figure 13: PCA Training Phase

Page 41: Report Final

34

Figure 14: PCA Recognition Phase

Page 42: Report Final

35

4.3.3 Calculation of Eigenfaces with PCA

4.3.3.1 Basic Definitions

Let a face image I(x,y) be a two-dimensional N * N array of 8-bit intensity values. An

image can also be considered as a vector of dimension N2, so that a typical image of

256 * 256 becomes a vector if dimension 65,536 or equivalently a point in 65,536

dimensional space. The basic idea of the principal component analysis is to find the

vectors that best account for the distribution of face images within the entire image

space. These vectors define the subspace of face images, which is known as “face

space”. Each vector is of length N2, that describes an N * N image, and is a linear

combination of the original face images. As a matter of fact, these vectors are the

eigenvectors of the covariance matrix corresponding to the original face images, and

because they are face-like in appearance, we refer to them as “eigenfaces”. Some

examples of eigenfaces are shown in Figure 15.

Figure 15: sample eigenfaces form the sample training set

Page 43: Report Final

36

4.3.3.2 Computation steps

For being acquainted with the terms and notations that are going to be used during the

eigenfaces algorithm evaluation, refer to Appendices A and B.

Step 1: Gather and prepare the data.

Let the training set of face images be T1, T2… TM

Step 2: Subtract the mean.

The average matrix � has to be calculated, then subtracted from the original faces

(Ti) and the result is stored in the variable � i.

The average of the set is defined by

Equation 4.3

Each face differs from the average by the vector

Equation 4.4

Page 44: Report Final

37

Step 3: Calculate the eigenvectors and eigenvalues of the covariance matrix.

This set of very large vectors is then subject to principal component analysis, which

seeks a set of M orthonormal vectors, Um which best describes the distribution of the

data. The Kth vector, Uk, is chosen such that

Equation 4.5

is a maximum, subject to

Equation 4.6

The vectors uk and λk are the eigenvectors and eigenvalues.

Step 4: Calculate the covariance matrix.

The covariance matrix C is calculated according to

Equation 4.7

Where the matrix A= [� 1,� 2,� 3…� M]

Page 45: Report Final

38

Following the matrix analysis, the M * M matrix L= A TA is constructed, where

Lmn=� Tm� n , and the M eigenvectors, V1 of L is computed. These vectors

determine linear combinations of the M training set face images to form the

eigenfaces U1.

Equation 4.8

The success of this algorithm is based on the evaluation of the eigenvalues and the

eigenvectors of the real symmetric matrix L that is composed from the training set of

images. After this step, the “training” phase of the algorithm is accomplished.

Page 46: Report Final

39

4.3.4 Classifying Images

Classifying an image with eigenfaces

A new face image (Γ) is transformed into its eigenface e components (projected onto

“face space”)

Equation 4.9

The weights form a feature vector,

Equation 4.10

describes the share of each eigenface in representing the input face image, treating the

eigenfaces as a foundation set for image images. The face classes Wi can be

calculated by taking the average of the results of the eigenface representation over a

small number of face images (as few as one) of each individual. Classification is

performed by comparing the feature vectors of the face library members with the

feature vector of the input face image.

This comparison is based on the Euclidean distance between the two members to be

smaller than a user defined threshold �k. If the comparison falls within the user

defined threshold, then face image is classified as “known”, otherwise it is classified

as “unknown” and can be added to face library with its feature vector for later use,

thus making the system learning to recognize new face images.

Page 47: Report Final

40

4.3.5 Input and Output

The inputs of our system consist of bit streams representing the image to be analysed,

an average face, an Eigenvector matrix, and a set of projections. The image to be

analysed, as well as the average face, will consist of 122 x 92 =10304 pixels, each

being 8-bit grayscale (28=256 levels ranging from 0 to 255). These figures were

chosen because they provide a good balance between size and accuracy. Additionally,

these values were used successfully by many research groups. In a database with M

faces, the Eigenvector matrix will be of size 10304 x M. Finally, the set of projections

will consist of M vectors, each having M values.

The output of our system will consist of the face ID with the closest match, as well as

the image from the database with which the closest match is found. All this

information will be displayed on the GUI of the PC. It will always call a stored

procedure in the database with image ID as its parameter to record the student’s

attendance in the database.

Furthermore, an email sent to the student’s registered email address confirming his

attendance for the lecture.

Page 48: Report Final

41

4.3.6 Matlab Implementation

The PCA algorithm was implemented on MATLAB and tested using AT&T ORL

face database. This database is composed of 400 images of 112 x 92 pixels each,

where these images represent 40 individuals with 10 different images for each

individual, representing various poses and facial expressions. A set of 4 images out of

each individual’s 10 images was taken as the training set, which was chosen as an

intermediate number to properly evaluate the performance of the algorithm. The

remaining 6 images of the individual were used for the testing of the recognition

phase.

4.3.7 Database Implementation

A dummy database was implemented on MySQL workbench to support recording the

attendance of the students. The actual database will be the university’s database which

contains real data of the students, courses, timetables and attendances records.

Page 49: Report Final

42

5 Discussion of Result

The initial phase of the project was faced by many challenges concerning the

materials and the methodology to be used. Building the project step-by-step or unit-

by-unit created concerns regarding the division of whole project into smaller modules

and the way of integrating them into a one complete working project. Separating the

design from its implementation was yet another challenge.

Division of the entire project into various sub-modules was done by recognising

various tasks needed to be performed. Each task was then handled with a different

script.

Integration of these sub modules was done basically by providing calls to and from

various method. The final GUI was so constructed that it handled the final integration

of both of the most important tasks, namely, Average Face Calculation and

Recognition. The constructed GUI also included the ways to access the information

stored in the database at real time along with calling functions for updating the

database.

It was noted that when a legal image was introduced to the system, 9.5 out of 10 times

it responded with the right match.

It was noted that if a match was found, the student’s email address was retrieved from

the database and an email was sent to his address confirming his attendance in the

particular lecture.

At present, the project cannot implement its tasks on an image with multiple faces.

Page 50: Report Final

43

6 Presentation of Results

6.1 The basic user interface

Figure 16: Basic GUI

This is the basic Graphical User Interface of the project for the user. Here the blank

area is to present the result upon clicking of any of the 2 push buttons provided.

Page 51: Report Final

44

6.2 Average face output

Figure 17: Average face output

The use of push button with text “Press to find the average face” gives the above

output. This is the calculation of average face from all the images present in the test

data.

Page 52: Report Final

45

6.3 Recognition output

Figure 18: Recognition output

Use of “Press for Recognition” push button prompts the user to enter the image.

When the user enters an image, its difference with the average calculated in the

previous step is calculated and checked for an image with the minimum variation of

this difference from the test data.

Page 53: Report Final

46

7 Conclusions

The working model of this project can perform all the intended tasks on the standard

face image database provided by the AT&T ORL.

The project is able to perform all the tasks described in the problem description with

sustained accuracy.

Though it is not yet ready for the real time, online processing of class attendances, it

is one of the optimal basic systems to provide a path for such a project.

Page 54: Report Final

47

8 Future Prospects

Some of the future prospects for this system are:

Processing real time, online data with capturing live images.

Integrating time table processing with this system to provide a single system

for a college for its 2 major functions i.e. scheduling of classes and marking of

attendance.

Integrating it with faculty attendance system by providing 2 different

databases.

Page 55: Report Final

48

9 References

[1] Alterman, A (2003). “A piece of yourself”: Ethical issues in biometric

identification. In: Ethics and Information Technology, Vol 5, pp: 139–150.

[2] Anhonen, T, Hadid, A, Pietikainen, M (2006). Face Description with Local Binary

Patterns: Application to Face Recognition. IEEE members

[3] Bennett, M.J (1996). Time and Attedance System and Method therefor. US Patent.

[4] Chalker, Jr, Spooner R.J(1982). Automated time and attendance systems. US

Patent.

[5] Cherry, K (2003). Biometrics: An In Depth Examination. In: InfoSec Reading

Room, SANS Institure. Version 1.4b, Option 1 pp. 3-23

[6] Graham, D (2003). It’s All About Authentication. In: InfoSec Reading Room,

SANS Institure. Version 1.4b, Option 1 pp. 2-12

[7]Jain, A.K., Pankanti, S, Prabhakar, S, Hong, L, Ross, A, Wayman J.L. (2004).

Biometrics: A Grand Challenge. In: International Conference on Pattern Recognition,

Cambridge, UK, pp 1-10.

[8] Jain, K, Ross, A, Prabhakar, S (2004). An Introduction to Biometric Recognition.

In: IEEE Transactions on Circuits and Systems for Video Technology, Special Issue

on Image- and Video-Based Biometrics, Vol. 14 No. 1 pp 1-29

[9] Li, S.Z., Jain, A.K. (2004). Handbook of face recognition. Springer Science and

Business Media, Inc. New York, USA.

[11] Maytas V, Riha, Z. Biometric Authentication- Security and Usability. Faculty of

Informatics, Masaryk University Brno, Czech Republic.

[12] Roethenbaugh, G (2005). Biometrics Explained. In: InterNational Committee for

Information Technology Standards.

[13] Tsalakanidou, F, Malassiotis , S, Strintzis, M.G. Face Recognition - Face

Detection, Global Approaches for, Feature Based Techniques, Problems and

Considerations, Conclusions and Future Developments.

http://encyclopedia.jrank.org/articles/pages/6741/Face-Recognition.html

[14] Zhao, W, Chellappa, R, Phillips, P.J., Rosenfeld, A (2003). Face Recognition: A

Literature Survey. In: ACM Computing Surveys.