[IEEE 2011 IEEE Colloquium on Humanities, Science and Engineering (CHUSER) - Penang, Malaysia...

Object with Symmetrical Pattern Recognition with Dynamic Size Filter

* Syed Mohamad Shazali Bin Syed Abdul Hamid, Mohd Shahrieel Bin Mohd Aras, Fara Ashikin Binti Ali, Fadilah Binti Abdul Azis , Mohd Anuar Bin Kassim

* Faculty of Electrical Engineering, UTeM, Hang Tuah Jaya, Melaka, Malaysia

Abstract— This paper presents the implementation of object with symmetrical pattern recognition algorithm for 2D vision system of 3D vision-based multi sensor feedback system. This paper also discusses the dynamic size filter developed for this system. The inputs for this system are received from a couple of webcams which is not calibrated. The project objective is to assist a robot arm system using vision system for making decision during conducting an operation or task. The solutions need no camera calibration because the system has self-calibration effect since the area of interest had modelled using a set of algorithms by an array of visible features on it and not by it environment. This system is design for on-event processes and suitable for indoor environment applications with low time frame rate camera. One of the advantages of this project is that it only used a camera and image processing generated by the algorithms itself without additional sensor such as sonar or IR sensor.

Keywords-Artificial Intelligence; Multi Sensor Systems; Robot Vision Systems; Size Filter

I. INTRODUCTION VISION systems have been used in industry as a single

sensor feedback to an industrial robot to detect the size, position and orientation of an object. The collected information will then be analysed by a computer to help the robot to understand some important features of the processed object and making any necessary decision.

As robotic and automation system is widely use and exposed to any kind of failure, input data provided by a computer vision system can greatly help the system to reduce their possibilities to fail and provide a lot of abilities to work on widely area of task and assignment. At the end, computer vision becomes a critical part of any artificial intelligence field or the industrial robotics in general.

Robot vision is a complex sensing process. It involves extracting, characterizing and interpreting information from images in order to identify or describe objects in environment. Vision systems should be able to acquire images, extract the relevant user-defined features just by processing the acquired images and return them through the available interface [ 1].

This project objective is to assist a robot arm system using vision system for making decision during conducting an operation or task. Katana 6M as a research robot for this project is controlled by a controller that will drive the single

arm industrial robot according to the input from the vision system. The kinematics analysis of the robot has been done earlier and discussed in [2].

The whole vision system, as describe in [3] consist of two web cameras but only one, which attached to the robot’s arm, pick the image of the inside of the boxes and its occupancies. The other one installed on a camera holder at the top of the working area, as shown in Figure 1. , providing top view of the hole working area. The arm will hold a camera that will capture the image of a cupboard consist of nine holes. Each holes have different properties in term size, distance between edges of neighbouring holes and different angle of each corner of the boxes. The other camera will take the image of balls on a pick table and feed the image to an image processor which will provide the objects’ coordinates in robot coordinate system.

Figure 1. The whole robot system setup

The whole design of the system built to enabling the robot to play Tic-Tac-Toe game. Two sets of balls with different colour have been chosen to represent Xs and Os and the holes representing all nine of the boxes involve in that game.

2011 IEEE Colloquium on Humanities, Science and Engineering Research (CHUSER 2011), Dec 5-6 2011, Penang

978-1-4673-0020-9/11/$26.00 ©2011 IEEE 696

This vision system has to provide not only the position of the balls in term of coordinate but also its current status of the game by viewing the face of the cupboard’s boxes that representing the board game of Tic-Tac-Toe. The captured image stored in the image processing load.

The different threshold value based on the hue saturated colour value was used. The idea is to use the two times threshold which is in saturation area and in hue saturated area. All cupboard’s boxes and the balls destination, and obstacle locations are determined using centre of area as a reference.

The algorithms used in this project designed to model the cupboard boxes’ occupancies and feed the information to the AI’s program in the robot controller for decision making process. It is also design to provide actual coordinate (in robot coordinate system) of the balls which are located on a nearby pick table. The main idea in this project is to aid the robot making right decision by enabling it to ‘see’ it’s working environment. The objective of this paper is to introduce the approaches and algorithms involved in modelling the cupboard boxes and identify the holes occupancies status.

II. APPROACH AND METHOD In the program will have three different difficulty settings:

easy, medium, and hard. For easy mode, the program will just place Xs randomly. For the mid-level difficulty, it'll be able to know when it's about to win and lose and play accordingly which is how most people actually play the game. For hard difficulty, it'll know the best way to play during any situation. A draw with this opponent is considered a perfect game.

In this project all the processes are running as on-event processes. This allow of using low frame rate camera. The processes are holes recognition, boxes position modelling, coloured balls detection, and PCS to RCS conversion.

A. Implementing AI Mainly there are two parts of the program that need input

from vision system; first, when the robot needs to know the current condition of the game or the user move and second, to verify the move that just been made by the robot.

The vision system should have the ability to recognize the holes in the boxes and model its arrangement pattern, identify ball in each hole, and give output to the AI program to which can be understood by the program. For that matter, a set of rules had been built to model the box’s holes and it contains.

The balls’ colour will be represented as different number as the holes will be represented as A to I. The output of this program will be presented and sent to AI routine function in a form as shown in Figure 2.

Figure 2. The standard form of holes status

B. Recognition of Coloured Object There are many ways to implement object classification

algorithms. In the previous researchers they are based on colour histogram [4], [5], motion detection [6] and template matching [7], but using colour to classify the objects is more practical compare motion or template matching. The reason is there is no need for other information from other type of sensory pressure or sonar, or to train the template for matching process.

To determine the colours in the image sequence, hue saturation value has been compensated the weakness of saturation value [5]. This is because by using saturation value, the different colours with the same range of the saturation are difficult to determine. Depending on this situation, two components are needed to determine the colours.

For detecting certain colour, the saturated and hue value need to be used as every colour has a different hue and saturated value. After the image of the front side of the cupboard has taken, the area with saturation value within the particular colour saturation value will be selected by selecting the pixels from the input image whose saturation values, s fulfil the following condition:

MinSaturation ≤ s ≤ MaxSaturation (1)

The value of the MinSaturation and MaxSaturation were selected so that all or most of the surface of every ball with a particular colour i.e. blue covered. All points of an image fulfilling the condition are returned as one region.

Afterwards, this region will pass another filtering process so that only the area with certain hue value corresponding to the particular colour within the region will be treated as possible surface of the balls. Small areas or large areas which are irrelevant to the balls in the region will be filtered out so that only the area with a reasonable size will be detected and treated as the balls with that particular colour.

Next, the detected coloured area transformed to rectangle shape depending on the smallest enclosing rectangle area followed by finding the centre point of the each rectangle. These points will be treated as the centre point of the balls in the next process.

C. Holes Recognition Holes usually can be identified by tracing the dark area

with certain grey value of threshold. After the image of the front side of the cupboard has taken, the dark area will be selected by selecting the pixels from the input image whose grey values, g fulfil the following condition:

MinGrey ≤ g ≤ MaxGrey (2)

The value of the MinGrey and MaxGrey were selected so that every hole covered and the shadow had filtered out. All points of an image fulfilling the condition are returned as one region. Small dark area in the region will filtered out so that only the dark area with a reasonable size will be detected and treated as holes.

[value box A, value box B, value box C, value box D, value box E, value box F, value box G, value box H,

value box I]


697

Next, the detected dark area transformed to rectangle shape depending on the smallest enclosing rectangle area followed by finding the centre point of each rectangle. These points will be treated as the centre point of the holes in the next process.

D. Size Filter For this operation, threshold operator is used to select

connected region built by connection region operator which have size within an interval that fulfilling it minimum and maximum size condition. All connected region in the image that fulfilling the condition are returned as possible object of interest. The possible value for the size must be choose properly so that it could filter out all or at least most of its possible noise.

The filter has two stages, namely as inner radius stage and box size stage. The inner radius stage will determines the largest inner circle radius of a selected region, i.e., the circle with the largest area of all circles that fit into the region. The series of value will be compared to threshold’s size and eliminate those regions which are not within the range. The area for this selected region then will be transformed to box shape for next stage of size filter.

The box size stage will filter out the boxes resulted from the previous stage that are outside the previously calculated threshold value, a. The value will be set by calculating the average of the boxes’ sizes. The average size value will be coupled with suitable range for each object which will be classified in a form shown in Equation 3.

AverageSize – RangeValue ≤ a ≤ AverageSize + RangeValue (3)

E. Modelling The Boxes The objective of this model is to enable the robot to classify

the status and features of the balls in each hole of the cupboard. For this process, fuzzy logic if-then rules had been used.

The rules built for this purpose is based on the intervals of predefine margin that had been calculated through Equation 4 and 6

(4)

where

(5)

and

(6)

where

(7)

as

Object coordinate in cupboard coordinate system

Object coordinate in pixel coordinate system

Centre point of holes coordinate in pixel coordinate system

had been chosen to eliminate or reduce error caused by the lens of the camera. The intervals, as visualized in Figure 3. , which represent the relative size of the cupboard and used to defines the location of the detected object (for this case a ball) in the cupboard’ boxes, is based on the cupboard model as shows in Figure 4.

0 10

A

20

B

30

C

10

D

E

F

20

G

30

H

I

Figure 3. Cupboard boxes model with fuzzy logic margin intervals.

Figure 4. The actual image of the cupboard boxes.

Whatever the size of the cupboard image in the captured image, the detected object coordinate in pixel will be converted into this relative cupboard coordinate. Through the application of Fuzzy Logic rules, the proposed system will figure the balls’ position and the boxes’ holes occupancies in the captured image.

The fuzzy logic rules applied for boxes modelling are represented as follow


698

If CPx<10 and CPy<10 then object in box A

If 10<CPx<20 and CPy<10 then object in box B

If CPx>20 and CPy<10 then object in box C

If CPx<10 and 10<CPy<20 then object in box D

If 10<CPx<20 and 10<CPy<20 then object in box E

If 10>20 and 10<CPy<20 then object in box F

If CPx>20 and CPy<10 then object in box G

If CPx>20 and 10<CPy<20 then object in box H

If CPx>20 and CPy>20 then object in box I

Where (CPx=Centre point in X-axis, CPy=Centre point in Y-axis).

Different colour ball will have different value to represent each colour such as orange as 1and blue as 2 as a blank holes will be represented as 0. Afterwards, the output of this program will be presented and sent to AI routine function in a form as shown in

III. RESULT In the interest of confirming the validity of the proposed

algorithms, an experiment was run using all the methods and approaches described previously. This section presents the experimental results of the analysis of the vision system. In this project, algorithms have been built for coloured objects detection, the cupboard’s holes recognition, cupboard’s holes’ occupancy and PCS to RCS conversion.

This experiment consists of two parts. The first part is a test on the output from the top camera (Camera 1) and the second part is a test on the output of the arm camera (Camera 2). The USB port communication is used in order to communicate between computer and camera.

The image of the front face of the cupboard with balls having three type of colour, blue, green and orange in its 8 out of 9 uneven size of holes had been taken with a webcam that had been placed at the arm of the research robot.

A. Recognition of Coloured Object This experiment done by image taken from the camera

placed on top of the research robot’s working place. The image taken includes blue, orange and green balls image. After applying colour filter, blue colour filter for example, all area with same hue and saturation value detected, as shows in Figure 5. To eliminate the irrelevant area, size filter had been applied and only area within the right size interval value treated as blue balls, as shows in Figure 6.

Figure 5. Detected blue area

Figure 6. Detected blue ball

B. Cupboard’s Hole Occupancy The image then processed using the process mentions in

Section II.C to extract information about the holes occupancy in meaningful form, as shown in

. With the intention of tracing the calculated value of the balls’ position in cupboard coordinate system, the program, which written by using an image processing program, Halcon, built to display the coordinate of each balls in cupboard coordinate system, as shown in Figure 7. All the steps in this experiment repeated as the number of the balls that occupied the holes had taken down to only one ball.


699

Figure 7. The output result of the object recognition in the boxes holes

The balls images are represented by X symbols with difference labels. This is because to ensure that we know which balls has been identified by the program. The result shows that the program successfully giving out the accurate result of the occupancy of the holes. However, at the beginning of the experiment, a few adjustments on the thresholds value had been done in order to eliminate the error that resulting detection of false image of balls as the initial threshold value fail to overcome the problems.

IV. DISCUSSION The systems are constructed from several algorithms; there

are coloured balls detection, holes recognition, PCS to RCS conversion and cupboard’s holes’ position modelling. The image processor is used to do all filtering process and integrate all the algorithms in a single system by using the image processing library and apply it to the algorithm.

The system is run in on-event processes and all information’s are appears in screen layout. Before an optional test, the right threshold value had to be selected to get the best hue saturated value. The threshold value is set by trial and error method with initial value based on the hue saturated and saturation value provided by software Paint. To make the threshold value to work with less affected by illumination, the threshold value had been widen so that all possible are with the certain colour can be covered. This is also the same with holes detection. The size filter, during the experiment successfully eliminates the unrelated area. However, when the balls are touched to each other which have same colour, the system is likely to consider the connected ball as a ball instead of two or three ball and sometime consider it as a noise as the size filter had narrow threshold.

For the cupboard modelling, the system is not in need of camera calibration setting as the cupboard modelling successfully modelled the cupboard without any of it. This is because the algorithm has some sort of self-calibration as the cupboard had modelled by the information of the holes of the cupboard and not by it environment.

Based on the consistency test result, the coloured object

algorithms manage to work in various light conditions with one condition which is the filter threshold value have to be change accordingly with light condition. Otherwise, the filter used in this algorithm will only work with success rate about 74% to 84%. However, considering the recommended range of luminance value by the IES for performance of visual tasks of high and medium contrast or large and small size is between 200 lux to 1000 lux [14], only the first two light conditions in the test scenario (scenario A and B) are within the range. The average of success rate for both scenarios is ranging from 87% to 100%. The algorithm, according to the results still could achieve average success rate more than 85% if working in light condition with more than 20 lux luminance, which is the lowest recommended light luminance condition for indoor activities.

This is possible because of the application of dynamic size filter. The filter allows the algorithm to choose appropriate size threshold value for different light condition. This enables the wider choice of threshold value for colour filter. This gives advantage to the algorithm to avoid filtering out actual image without reducing its credibility to filter out noises.

V. CONCLUSION The objectives of this research are achieved. The vision

system in this project managed to recognize objects with different colour from two sources of image, top camera and on arm camera. The vision system enables to recognize the different colour of the objects and provide the coordinate of the object. The system also could recognize the relevant condition of the objects’ position in its environment.

One of the advantages of this research is that it only uses a low resolution camera and image processing software generated by the algorithms itself without additional sensors such as sonar or IR sensor. This research developed an improved technique on object recognition and space occupancies which not affected by the orientation of the subject. This project also implements coloured object recognition technique by its colour and size without edge detection process along with a self-calibration technique for detecting object location without any parameter of the camera by using only two reference points.

This research intentionally working with low resolution camera, which is relatively cheap compare to high resolution camera, along with commercial image processing software, with inspiration to promote the usage of robot operator instead of human operator in industries by developing inexpensive supporting system for industrial robot.

ACKNOWLEDGMENT The authors gratefully acknowledge the information and

cooperation form Professor Marizan Sulaiman, Mohd Azmi Said, Hairol Nizam Mohd Shah, Associate Porfessor Dr Zulkifli, Professor Marizan Sulaiman and the researchers at the Faculty of Electrical Engineering, Universiti Teknikal Malaysia Melaka (UTeM) for their relevant research in robot vision system.


700

REFERENCES [1] Pire, JN, Loureiro, A. and Bolmsjo, G.,. Welding Robots. London :

Springer-Verlag, 2006. [2] Analysis of Kinematics on Katana 6M Robot. S. M. Shazali S. A.

Hamid, Marizan Sulaiman, M. Azmi Said. Melaka, Malaysia. : s.n., 2007. Proceedings of the International Conference on Engineering and ICT. pp. 803-807.

[3] Multi Sensor Vision System for Robot System with Artificial Intelligent. Syed Mohamad Shazali, Syed Abdu Hamid, Marizan, Sulaiman and Azmi, Said. Pulau Pinang : UiTM Pulau Pinang, 2008. International Conference on Science & Technology:. pp. 1295-1301. ISBN 978-983-42204-1-9.

[4] Ching, H.K and Wen, H.T. Obstacle Avoidance in Person Following for Vision-Based Autonomous Land Vehicle Guidance Using Vehicle Location Estimation and Quadratic Pattern Classifier. s.l. : Dept. of Computer and Information Science. National Chiao Tung University,Taiwan, 2001.

[5] Hideki, H. Intelligent Space for Secure and Active Humans’ Life. . s.l. : Institute of Industrial Science, University of Tokyo, 2004.

[6] Dedeoglu, Y. Moving Object Detection, Tracking and Classification or Smart Video Surveillance. s.l. : Institute of Engineering and Science, Bilkent University, 2004.

[7] PerS and KovaEiE. Computer Vision System for Tracking Players in Sports Games. s.l. : Faculty of Electrical Engineering, University of Ljubljana, 2002.


701

[IEEE 2011 IEEE Colloquium on Humanities, Science and Engineering (CHUSER) - Penang, Malaysia...

Documents

Transcript of [IEEE 2011 IEEE Colloquium on Humanities, Science and Engineering (CHUSER) - Penang, Malaysia...