3rd Year IGEN Project - Final Report

IGEN 330 Final Report

Submitted April 7, 2015

Zefan Sramek, Jasmin Liang, Ryan Corkery, Gurjot Bal, Michael Harvey, Rahim Moosa,

Darshan Soni

Executive Summary

Optical quality assessment, more commonly known as Automated Optical Inspection

(AOI) [Glossary], is used in a variety of industries. The printing industry faces considerable

losses due to errors in the printing process itself. The aim of this project is to develop a cost

effective system to detect errors in the printing stage and allow for corrections to be made

sooner. The primary specifications of our project are that it must be able to detect errors up to 0.5

mm small and be able to cover an area of 32” X 41”. A system comprising of an array of two

cameras, a three piece mount and a laptop running on linux were chosen to meet the design

goals.

With the various specifications in mind, the project was then split into three design

subgroups: camera array, camera mount, and software. The camera array is responsible for the

detection, capturing and uploading of images of the moving labels with potential flaws. These

design goals are met by using a development kit associated with the cameras, and uses built in

commands to accomplish these functions.

The camera mount is responsible for designing an arch that is able to safely hold and

stabilize the cameras, and be resizeable for various sizes of printed operations. The chosen

design is dimensioned in SolidWorks and is built by hand using ABS [Glossary] piping.

The software is responsible for aligning and analyzing the captured photos for any errors.

If any errors are found in the photo, a notification is sent to the worker. All of this is

accomplished through scripting in C++ and using the library OpenCV [Glossary].

A prototype was assembled with these subgroups. The testing of the prototype, and its

results are documented below. Future improvements to the prototype are investigated.

Table of Contents

1.0 Project Design

1.1 Needs Statement

1.2 Problem Specifications

1.3 Initial Design

1.4 Final Design

1.4.1 Camera Array

1.4.2 Software

1.4.3 Camera Mount

2.0 Assembly and Commissioning

2.1 Prototype Assembly

2.2 Production Model Assembly

3.0 Project Testing and Modifications

3.1 Individual Testing

3.2 Overall System Testing

3.3 Results

3.4 Modifications

4.0 Future Work

5.0 Conclusion

6.0 Appendices

7.0 Glossary

8.0 References

Table of Figures

Figure 1: Flowchart of the process

Figure 2: Representation of two different camera arrays

Figure 3: Geometric model of the mount done using SolidWorks

Figure 4: Camera motion detection grid, 6X6


Figure 6: With error on right leg of Koala

Figure 7: Identical photos with different brightness and associated error

Figure 8: Images in grayscale outlining edges and errors

Figure 9: Different levels of Pixel brightness are converted to completely black or white

Figure 10: All errors are detected and are present in white

Figure 11: Typical Input Image

Figure 12: Corners identified including inner corner

Figure 13: Result After Alignment

Figure 14: Slight Variations Detected

Figure 15: Label with error detected

Figure 16: Error detection of tested label

Figure 17: Initial stand drawings

Figure 18: Dimensions of the frame

Figure 19: Tested label placed on a black poster to detect edges and corners

Figure 20: Testing done on tripod

Figure 21: Black poster board background with white labels being used for testing

Figure 22: Black cloth background with white labels being used for testing

1.0 Project Design

1.1 Needs Statement

The aim of this project is to develop a cost effective system to detect errors in the printing

stage for silk screen printers and allow for corrections to be made sooner. The label creation

process includes several stages after the printing stage, most of which are more complicated and

more costly than printing. Implementing an automated optical inspection (AOI) [Refer to

Glossary] system into the screen printing process will allow flaws to be detected before these

later stages. Production time is reduced if prints with errors are removed from the production line

immediately and feedback is given so that the errors are not replicated. The product will cut

production and operating costs by replacing manual error detection at the end of the line, with

automated optical inspection at the beginning of the line.

1.2 Problem Specifications

To gain more insight on the specifications of the project, a tour was conducted at Ampco.

Ampco is currently MetroVancouver’s largest printing company, and provided relevant

specifications relating to printed labels. The majority of products printed by Ampco are to their

internal A standard, as described in Visual Quality Inspection for Overlays, Panels and Graphics

[1]. The specifications for error detection come from this standard. No more than a single error is

permitted per item. The only permitted errors are dots with a diameter of less than 0.5mm and

dust particles with a diameter of less than 0.5mm [Table 1]. The most common type of error

seen, and the hardest to detect by eye are dots, pin holes, and dust particles. As a result the

primary error detection specs are:

1. Detect the presence of multiple errors on a single label on a multilabel sheet

2. Detect the presence of dots and pin holes of 0.5mm diameter or greater

3. Detect the presence of dust particles or their results of 0.5mm diameter or greater

Beyond these error detection specifications, other specifications are necessary to be met

in order to create a system that will operate efficiently with Ampco’s current printing method.

The majority of errors occur during the screenprinting stage of label production. The error

detection system must, therefore, interface with the printer used by Ampco. The primary

integration specifications are:

1. Process images at the same rate as prints are produced (approximately 1 image per 8 seconds)

2. Selfsupport in such a way as to not interfere with the printing process

3. Process prints up to a maximum size of approximately 32” x 41”

Once the system detects an error, it is necessary to remove that print. Since most errors

are caused by a dirty screen with residue ink[2], screen cleaning is required. The cleaning is to be

done by a worker.

1.3 Initial Design

Upon examining the needs and specifications, the problem was broken down into 5

categories or stages:

1. Acquire Ideal Label for Comparison (Optional)

2. Capture Label for Processing

3. Mount Capture Technology

4. Process Image to Detect Presence of Errors

5. Notify Worker of Error

Figure 1: Flowchart of the process

Multiple methods to detect labels, mount the system, and to analyze the labels were

investigated. These included which types of cameras to use, the shape and material needed for

the mount, and what system was required to run the code for analysis.

By comparing various aspects of three areas of interest (camera type and array, mounting

hardware, computer hardware) using design matrices, it was finalized to use a system comprised

of two Canon PowerShot ELPH130IS cameras to capture the image, a three piece arch to mount

the system, and a laptop running on Linux to process the image[3].

Figure 2: Representation of two different camera arrays

Figure 3: Geometric model of the mount done using SolidWorks

1.4 Final Design

1.4.1 Camera Array

The category of the camera array relates to the two Canon PowerShot ELPH130IS

cameras and their functionality. With the cameras in mind, three design goals were set out:

1) Be able to consistently detect a moving label, and notify the system when the entire label

is in range of the camera.

2) Trigger both cameras at the same time, and stitch the photos together to create an image

capturing the entire label.

3) Upload the image to a computer, and make it accessible to analyze.

Initially, there were some issues, as the means to program the cameras, Canon’s source

development kit (SDK), was not available for the point and shoot models. A possible solution

was to change the scope of the project, and use a single digital singlelens reflex (DSLR)

[Glossary] camera which would be supported by Canon’s SDK. These cameras however were

priced at a minimum of $450, and were not deemed suitable for the project and its limited

budget.

Through further research, an unofficial development kit, the Canon Hack Development

Kit (CHDK) [Glossary], was found that accomplishes the same tasks as the official SDK, but

supports point & shoot type cameras. The CHDK is loaded onto an SD card, and inserted into the

camera, enabling the user to upload scripts to the camera. An extension application, the Picture

Transfer Protocol Cam (PTPCAM) [Glossary], is also used to access CHDK’s functionalities

remotely with a USB cable and computer.

In the first week, work was focused in triggering both cameras remotely. A script was

written in Bash that incorporated both PTPCAM and the CHDK commands to loop the cameras,

and made them shoot in synchronization. The following week, the script was edited to upload

and delete photos off the camera to a given directory by using the PTPCAM extension.

To detect moving labels, research was done involving different types of sensors. It was

found that the cameras had built in Complementary metal–oxide–semiconductor (CMOS)

sensors, that work as colour sensors, by capturing light and converting it into electrical signals.

The CMOS sensors were chosen, as they accomplished the main goals and fit within the budget.

A script was written in bash that incorporates the sensors.

The camera’s image range is split up into a array (6X6 for chosen design) of squares

[Figure 4].


Each square represents a portion of the entire image with an associated threshold

sensitivity value. Since the camera can’t actually detect motion, it instead detects the change in

pixelation. When an object moves past the camera, the colour and thus pixelation will change.

The CMOS sensor picks up this change of colour and assigns a threshold sensitivity value to the

rate of which the colour is changing.

With these aspects in mind, a singular column of the 6X6 array is chosen to be the

‘activation zone’. In the ‘activation zone’, a parameter is set to the threshold sensitivity value,

and will trigger the cameras if this parameter exceeds this value [Figure 5]. After testing, and

finding a numeric value suitable to detect labels entering the ‘activation zone’ [Glossary], the

script was finalized and incorporated with the software.


1.4.2 Software

The specific goals of the software side of the project were to be able to detect dots and

pinholes, the presence of dust particles, and to be essentially unaffected by the layout and design

of the labels being printed. In essence, the goal was to detect the presence of deviations from a

set design, known at the onset of printing. Seeing as the initial label design was to be set from the

beginning of the printing, it was determined that the most efficient method by which to detect

errors would be to compare between a “perfect” image and the photograph of a newly printed

label.

The first step in processing a label for defects is to check that a photo of the label was

taken and saved onto the computer. The system does this by looking into the specified computer

directory and checking if a file exists. After analysis, the image is deleted from the directory

before the next photo taken is saved into the same directory.

It was necessary to determine a method for processing the images that would include the

required tools and would execute in a limited time. The C++ library OpenCV was chosen

because it was designed precisely for the type of task required for this project and works with the

fastexecuting C++ language.

The first goal in the development of the errordetection software was to detect a

difference between two otherwise identical images. In OpenCV, and more generally in digital

systems, images are represented as matrices of colour data. For a standard colour image, the

matrix contains vectors with three values, one for the red, the blue, and the green values that

make up the colour of each pixel. Thus, the preliminary algorithm simply iterated through the

matrices of the “perfect” image and the test image and marked any pixels with different colour

values in bright green [Figure 6].

Figure 6: With error on right leg of Koala

This method was effective, but only for images that were mathematically identical in

every way except for the defect. This meant that the same image with different brightness levels

would result in an error being detected at every pixel [Figure 7].

Figure 7: Identical photos with different brightness and associated error

The solution was to simplify the image and remove unnecessary data. To detect a defect, it

was necessary to detect a sudden shift in colour. The colour itself, however, was unimportant. The

types of labels being scanned were likely to be simple and blocky, with little intricate detail. These

facts allowed a new, more robust algorithm to be developed. Since the colour of the images was

irrelevant and sudden changes in colour needed to be detected, a method making use of edge

detection algorithms became apparent as a possible solution.

The first step of the algorithm is to apply a Gaussian blur [Glossary] to remove noise and

convert the images to grayscale [Glossary]. This is achieved using OpenCV functions. Next the

gradients of each image are taken in the X and Y directions and stored into matrices using the

Sobel operator [Glossary]. When these matrices are overlayed, they generate an image in which

the edges of colour regions are white while the rest of the image is black [Figure 8].

Figure 8: Images in grayscale outlining edges and errors

Once this operation is complete, the errors become particularly apparent to the human eye.

However, the simple comparison algorithm still has difficulty identifying only the errors. The

brightness of the pixels tracks the actual gradient of the image, and is still affected by the original

images’ brightness differences. As a result, further processing is necessary.

To deal with the different levels of pixel brightness, a threshold is applied to the images,

colouring the pixels either completely white or completely black [Figure 9].

Figure 9: Different levels of Pixel brightness are converted to completely black or white

As visible above, the images still contain a fair bit of noise at this stage, and thus still result in

imperfect error detection when compared. This final stage is to remove the noise present in the

image resulting from the comparison by subtracting the completely black and white “perfect”

image from the result and making any negative values black. This results in an image where only

the errors are visible in white [Figure 10].

Figure 10: All errors are detected and are present in white

This method of error detection is still affected by deviations in alignment of the test

image. Naturally, unavoidable shifts in alignment with the camera will throw the algorithm off.

A typical input image to the system from the camera includes the label itself and a large portion

of background [Figure 11].

Figure 11: Typical Input Image

In order to allow the comparison algorithm to work, the background must be removed

and the label must be aligned. This perspective shift and cropping can be achieved using another

function of OpenCV. This function takes a set of four xy coordinates from one image and maps

them to another set of four xy coordinates on another image. The more complex step involved in

the alignment process is the locating of the corners of the label in the input image. The corner

detection is achieved using the OpenCV implementation of the Harris corner detection

algorithm [Glossary]. This algorithm works by calculating the change in pixel values over an

area and identifies a corner when there is a significant change of colour in multiple directions.

Running this algorithm often identifies multiple coordinates for each corner and thus the next

step is to remove duplicate points. This is achieved by identifying points within five pixels of

each other and removing all but one.

Figure 12: Corners identified including inner corner

Once only unique points are left, it is necessary to identify the points that are the outer

corners of the label. Often other corners are identified within the image on the label and these

must be disregarded [Figure 12]. This is done by calculating the distance from the centre of the

image to each corner and taking the 4 points with the largest distances as the four outer corners.

This techniques assumes that the background is smooth and free of any sudden changes in colour

or shade. Once the outer corners are identified, they are sorted based on their position relative to

the quadrants of the original image and sent to the function to apply the alignment, as described

above [Figure 13]. Once the alignment is complete, the images are sent to the comparison

algorithm.

Figure 13: Result After Alignment

Although the alignment technique makes it possible to compare images with skewed

labels, the comparison algorithm as described above still results in the detection of non erroneous

label features due to slight variations in the exact pixel locations of the corners [Figure 14]. Thus

the output of the original comparison algorithm does not accurately identify only errors.

Figure 14: Slight Variations Detected

In order to identify the areas that constitute real errors, a blob detection algorithm is run

on the output image [Figure 15]. This OpenCV function identifies large regions of neighbouring

identical pixels and outputs their coordinates. Applying this algorithm with a threshold blob area

of fifty pixels and blob colours of black or white allow only true errors (blobs of large enough

area) to be detected and identified with red circles on the original image [Figure 16].

Figure 15: Label with error detected

Figure 16: Error detection of tested label

1.4.3 Camera Mount

The camera mounting hardware went through several design iterations, as the

requirements of the mount changed. Through the use of a design matrix, the first design was a

large steel frame that offered a large degree of adjustability [Figure 17]

Figure 17: Initial stand drawings

As it became clear that the product would not be delivered to Ampco, it was decided that

the frame no longer needed such a large investment. A cheaper design that could be built by hand

was investigated. The first component considered was the material of the frame. ABS piping

replaced steel as the material for the frame, due to ABS piping’s lower cost and ease of

manufacturing. The camera housing was initially one large box intended to hold both cameras

and the components of a desktop PC. The desktop PC was removed from the design and replaced

with a laptop, reducing the components in the box to only the two cameras. The best option was

then to use two small boxes rather than one large box.

Figure 18: Dimensions of the frame

2.0 Assembly and Commissioning

2.1 Prototype Assembly

The prototype was manufactured manually. This is the most effective method because of

the integration between hardware and software components. The hardware assembly uses simple

ABS piping for the stands as it is readily available in different sizes and is easy to process. The

piping is cut to stand measurements. The software is assembled by simply integrating the two

parts of the code (camera & picture analysis scripts) into one executable file on the laptop.

2.2 Production Model Assembly

The mass production model will be very similar to the prototype but the ABS piping will

be replaced by a simple steel framework with adjustable slots. Steel will give the frame the

durability required to last in an industrial environment.

The 3 piece arch will be held together with large pins. They are capable of holding the

frame in place for long periods of time, but are also easily adjustable.

With each aspect of the project covered, each part was assembled into the initial

prototype, and testing was then done.

3.0 Project Testing and Modifications

3.1 Individual Testing

Initial testing was done to determine the reliability of the software, consisting of only the

camera triggering code and error analysis/alignment. To test the image analysis and alignment

code, photos of various labels were taken, some with errors, on top of a black poster board. A

black poster board was used as a background to allow easier detection of the edges and

corners [Figure 18].

Figure 19: Tested label placed on a black poster board to detect edges and corners

However, there were problems that were encountered when the code was executed. The

corner detection script would find less than 4 corners, and therefore was not able to realign the

image correctly. It was thought that these issues were caused because:

a) The background was not dark enough and so the contrast of the white label to the black

background was not sufficient for edge detection.

b) Images were taken by hand, throwing off the focus of the camera and making the

image blurry.

To get around this, testing was then done using a tripod to hold the camera, setting an

exact range for the focus through CHDK, and by spray painting the poster black to emphasize

the contrast [Figure 20].

With the updated testing conditions, the 4 corners of the label could be detected, however

not consistently. This is believed to be the result of unevenness in the spray painting, leading to

different shades on the poster board. Albeit the inconsistency, the testing showed that the image

realignment script worked as intended.

Testing involving the label detection was also done to determine the reliability of the

code. The camera was set on the tripod facing a black background. A piece of paper held by hand

was then passed by the camera to simulate a label moving down the production line. As

expected, once the paper passed by the activation range [Glossary] of the camera, the camera

triggered, took a photo, and uploaded it to a given directory on the computer system.

Figure 20: Testing done on tripod

3.2 Overall System Testing

To test the overall system, the physical and software components of the project were

combined into one system. All components of the software including camera triggering, image

analysis, and image alignment were combined into an executable file to be run on a laptop. The

camera was mounted on the three piece arch a vertical distance away from a table [Figure 21].

The camera was then triggered remotely for error analysis and alignment. Some problems were

encountered in the testing of the system:

Camera focus: Focus was an issue as the alignment script was very sensitive, and would

be thrown off if the image was not perfectly focused. The camera’s autofocus was not deemed

suitable and instead a manual focus command was used. By measuring the distance between the

table and camera, the exact focal distance could be set in the code. This allowed the camera to

take a clear, sharp image of the label that was suitable for error detection.

Corner detection: As in the individual component testing, a recurring problem was that

less than 4 corners were detected. In attempt to improve the results from individual testing, a

black plastic table cover was used to lay the labels on. This was not ideal as parts of the black

cover displayed glare which possibly interfered with error computation. A black cloth cover was

then used instead, and as expected, the errors in the label were consistently detected. The

extreme contrast of the black cloth and the white label was the solution to the problem.

Figure 21: Black poster board background with white labels being used for testing corners not

reliably detected

Figure 22: Black cloth background with white labels being used for testing corners detected

reliably

3.3 Results

Through testing, it was shown that the system was able to consistently pick up most

errors in simple images to a max size of 18” by 18”. The label detection and error analysis code

were able to work together to provide consistent results, and the mount was able to safely house

the entire system while stabilizing the cameras to provide clear photos.

In the error analysis, only the approximate area of the errors are labeled [Figure 16].

Tests were done using a variety of different labels with sample errors drawn on, and proved to

consistently detect and label said errors.

There were some issues relating with the PTPCAM extension and file uploading. When

the photo is uploaded, the cameras must reboot before they can start detecting again. These

reboot times would take only approximately 8 seconds, so would still meet the specifications of

the project.

3.4 Modifications

Modifications and additions are required in the software. The initial specifications of

detecting labels 32” by 41” were not met. To meet this range, additional coding would be

required to stitch two photos together. Other modifications to the code would be to work on the

error detection, as the current code shows only the approximate area of faulty pixels.

4.0 Future Work

Due to time constraints and complications with image analysis, the prototype was not

able to meet the specification of detecting labels 32” by 41”. Future work would involve creating

an image stitching script that utilizes two cameras to capture a bigger area. Work would also be

put aside to develop a more reliable code to detect errors.

Using CHDK, tethered shooting could be supplemented with live view on a computer

monitor. This would allow the user to see the image of the label and to adjust the focus before

the camera is triggered. This would increase the reliability of the error detection by reducing

unfocused label images.

5.0 Conclusion

The overall scope of the project changed during design. Various design aspects, such as

the number of cameras used, what computer system to use, and the material of the mount

changed during the design. These changes were necessary to meet deadlines and budgetary

constraints. Overcoming these design challenges helped develop problem solving skills.

Project design skills were also developed throughout the course of the project. Relevant

skills learned include the ability to study a market, determine whether the market is suitable to

enter, and how to develop a project to meet specifications effectively.

Other skills involve teamwork skills. These include being able to effectively

communicate individual progress to the group, splitting up work evenly and efficiently, and

having a general scope of each role.

Overall, the project goals of the project were accomplished by creating a device that

detects errors in printed labels. Although the system did not turn out exactly as envisioned, the

final label inspection system is able to take a photo of a label, align the label properly for error

detection, and detect errors in the label.

6.0 Appendices

Cost Breakdown

Bill of Materials

ABS piping $59.76

ABS joints $28.52

Nuts and Bolts $11.04

ABS cement $6.20

Cameras $252.97

2 SD Cards $22.01

Black table cloth $10.02

Black paint $16.78

A fully functioning AOI system Priceless

Total $407.30

Table 1: Ampco’s Class A Visual Quality Standard for labels

Table 2: Program Flow Chart

7.0 Glossary

ABS (Acrylonitrile butadiene styrene) Amorphous thermoplastic polymer that is low cost and

resistant to heat, chemical, and impact. Often used for appliance housing, luggage, camera bodies, and

various furniture components.

Activation Range During motion detection, the camera’s image is separated into a grid system.

The activation range is the selected squares off the grid that when triggered will cause the camera

to shoot. This is to provide more control over the process and trigger the camera only when the

entire image is in range.

AOI (automated optical inspection) A system using a camera to scan an image for

manufacturing errors and quality defects.

CDHK (Canon Hack Development Kit) An unofficial development kit, similar to Canon’s

SDK but aimed at point and shoot cameras. The downloaded CHDK file is loaded onto a SD

card and inserted into the camera, enabling the user to upload scripts and extensions to the

camera.

DSLR (Digital Single Lens Reflex) A digital camera with a digital imaging sensor and

interchangeable lens. Images are typically of higher resolution and quality than images taken

with a point and shoot camera.

Gaussian Blur Blurring of an image using a Gaussian function to reduce image noise and

detail. Most edgedetection algorithms are sensitive to noise, so this preprocessing stage

enhances image structures at different stages and improves the result of the algorithm.

Grayscale A range of colourless shades in gray.

GUI (Graphical User Interface) A program interface that allows users to interact with

electronic devices though graphical icons and visual components. Instead of triggering camera

from terminal, GUI would allow worker to click on one icon to take a photo of the label.

Harris Corner Detector A function used in image processing that detects corners by detecting

variations in pixel values in an image matrix. Corners are detected when there is significant

change in multiple directions in a small region of pixels.

OpenCV An open source library of C/C++ computer programming functions aimed at

realtime computer vision (methods for acquiring, processing, analysing, and understanding

images).

PTPCAM An extension that allows the computer to send commands to the camera via CHDK.

Sobel Operator A function used in image processing and computer vision within edge

detection algorithms to create an image with defined edges and transitions. By computing an

approximation of the gradient of the image intensity function, the result of the Sobel operator is a

2dimensional map of the gradient at each point.

8.0 References

1. Ampco, Visual Quality Inspection for Overlays, Panels and Graphics

2. Tour with Brian Fewtrell Sr. Manager of Engineering & Quality October 6th 2014

3. Interim Report 1

4. OpenCV Online Documentation : http://docs.opencv.org/

3rd Year IGEN Project - Final Report

Documents

Transcript of 3rd Year IGEN Project - Final Report