1ECE 495 – Integrated System Design I

- Ninad Pradhan, Matt Pepper

Introduction to Image Processing

ECE 495, Spring 2013

2ECE 495 – Integrated System Design I

Overview

• What is image processing?• What is an image?• Introduction to image processing techniques• Experimental setup• Basic MATLAB commands for image processing

3ECE 495 – Integrated System Design I

What is Image Processing?

Image processing is the collective name for techniques used to extract information from digital images or to manipulate them to

render variations of the input image.

Photo stitching Color boost

Vehicle detection and tracking

4ECE 495 – Integrated System Design I

What is Image Processing?

Popular technologies which make use of the camera as a sensor

The Wii Remote uses an IR camera to sense its location relative to the

Wii Sensor Bar.

The Kinect uses image processing techniques on depth images to detect

and track locations of multiple persons in the field of view.

5ECE 495 – Integrated System Design I

Pixels

Pixel

• A pixel (abbr. for picture element) is the smallest unit of an image.

• Therefore, a 640x480 image is a matrix of 640 columns and 480 rows, each element of this matrix is called an image pixel.

6ECE 495 – Integrated System Design I

MATLAB Image Coordinates

• MATLAB stores images as matrices.• In MATLAB, image pixels are referenced using (row, col)

values.• Origin of the coordinate system (1,1) is the top left corner of

the image

img

Thus, img(4,3) refers to the pixel at the 4th row and 3rd column.

(1,1)

7ECE 495 – Integrated System Design I

RGB and Grayscale

• In RGB format, each Pixel has 3 color components: Red, Green, and Blue.

• Other color representations, e.g. HSV, YUV, CMYK, are also used. Transformations from RGB to these color spaces and back are defined in MATLAB.

• If only intensity (bright/dark) variations are considered, the resultant image is called a grayscale image. Each pixel has only 1 component: intensity.

RGB Gray

8ECE 495 – Integrated System Design I

Basic Image Processing techniques

• Background Subtraction• Intensity Thresholding • Morphological Operations• Color-based Processing• Shape-based Processing

A combination of the above methods is usually used in basic image processing applications. There is no ‘perfect solution’, so keep discussing (and trying) what suits your project best.

9ECE 495 – Integrated System Design I

Background Subtraction

• Background subtraction is a technique used to isolate useful information in an image (foreground) from the rest of the image (background).

• A reference image is selected as the background.• Each successive image in a video stream is compared against

this image.• If the difference between the images is significant, the areas

which are different are considered to be the foreground for that image.

10ECE 495 – Integrated System Design I

Background Subtraction

Reference image or background

Incoming image contains a new object. This is our foreground, i.e. useful information for detecting the system state.

The new object is identified as the foreground after the background is subtracted.

11ECE 495 – Integrated System Design I

Intensity Thresholding

• Intensity thresholding is another foreground separation technique. It uses histograms.

• A histogram is an image statistic which usually operates on an intensity image, i.e. pixels having a single value between 0-255

• The range 0-255 is divided into bins, e.g. each intensity value may be given its own bin

• The Y axis shows the count of number of pixels in an image which lie within limits for a bin.

Bin

Cou

nt

12ECE 495 – Integrated System Design I

Intensity Thresholding

• Assumption: Foreground is sufficiently bright/dark compared to the background, i.e. the histogram has a clear separation.

• Assumption: The threshold value has been estimated using prior observations or can be computed automatically.– This method is very sensitive to lighting conditions

• Given these assumptions, set all pixels on the foreground side to 255 (bright), and all other pixels to 0 (dark).

Foreground

Threshold

13ECE 495 – Integrated System Design I

Morphological Operations

• Morphological operations are used to either remove noise from an image or recover some details lost in thresholding or background subtraction.

• Dilation adds to the foreground areas in an image, but can also add to the noise.

• Erosion removes stray foreground pixels from it, but also reduces genuine foreground areas.

• Usually, a combination of morphological operators is used to improve the output.

After dilation After erosionOriginal

14ECE 495 – Integrated System Design I

Color Processing

• Used in identifying well color for the Minesweeper problem.• In MATLAB, a color (RGB) image is stored as a 3D matrix with

the third dimension being color.• Each pixel has a Red, Green, and Blue value. The values range

(for each color) from 0 to 255, e.g. a purely red pixel at (25,30) will have the RGB value (255,0,0).

• To access the Red value for this pixel in the 3D image matrix, we reference img(25,30,1). Similarly, Green will be img(25,30,2) and Blue, img(25,30,3).

img

15ECE 495 – Integrated System Design I

Color Processing

• Color processing algorithms need to have a tolerance to color, e.g. a RED sticker won’t have RGB (255,0,0).

• This tolerance can be built into the algorithm by – Testing under varied ambient lighting

conditions, e.g. morning, evening, and setting color thresholds accordingly.

– Experimenting with more variation-tolerant color spaces, e.g. HSV.

– Experimenting with logical operators, e.g. a RED sticker = (Red enough) AND (NOT Green enough) AND (NOT Blue enough).

– Take average of pixel values in area

Is there a red sticker in this well?

RGB (173,58,86)

16ECE 495 – Integrated System Design I

Shape-based Processing

• Once the foreground objects have been detected, it is important to know their properties such as area, shape, location, etc.

• MATLAB has a function called regionprops which takes a binary image as input and identifies properties of contiguous (connected) pixels in foreground regions.

• The output of regionprops is an array of structures. Each element of the array contains information about one foreground region.

• MATLAB help file is very usefulRegion 1

Region 2Region 3

17ECE 495 – Integrated System Design I

Shape-based Processing

Regionprops can be used to filter out poor foreground detection results. For example:

1

2

3

4

56

All detected foreground regions

Eliminate those which are too small [5]

Eliminate those which are not circular [6]

Eliminate those which are not at permitted

locations [4]

Genuine foreground detections

1

2

3

[1 2 3 4 5 6]

[1 2 3 4 6]

[1 2 3 4]

[1 2 3]

18ECE 495 – Integrated System Design I

Experimental setup

When the camera is set up in a known configuration relative to the rest of the system, we can estimate physical dimensions using image data. For example:

Camera field of view

Fixed height 480

pixels

640 pixels

16”

12”

This implies• 40 pixels/inch

along image width

• 40 pixels/inch along image height

A circle 1.25” diameter will be bounded by an area of:

1.25” * 40 pix/inch * 1.25” * 40 pix/inch = 2500 pixUse information to improve game state detection , criteria for algorithms

Width Height

19ECE 495 – Integrated System Design I

Some MATLAB functions

Name Functionality

imread Read input image

imshow Show the image in a figure window

rgb2gray Convert a RGB image to grayscale

rgb2hsv Convert a RGB image to HSV

imdilate Dilation

imerode Erosion

im2bw Convert image to binary using thresholding

regionprops Get properties of connected components in an image

imhist Show image histogram