C#, OpenCV, and simple Camera/Robot Calibration

Sebastian van DeldenUSC Upstatesvandelden@uscupstate.edu

OpenCV

Open Source Computer Vision Library OpenCV is Intel’s Open Source Computer

Vision Library. It is a collection of C functions and a few C++

classes that implement some popular Image Processing and Computer Vision algorithms.

EMGU provides wrapper class packages so that OpenCV can be used in C#.

3

Features Image data manipulation

allocation, release, copying, setting, conversion Image and video I/O

file and camera based input, image/video file output Matrix and vector manipulation, and linear algebra

routines products, solvers, eigenvalues, SVD

Various dynamic data structures lists, queues, sets, trees, graphs

Basic image processing filtering, edge detection, corner detection, sampling and

interpolation, color conversion, morphological operations, histograms, image pyramids

4

Features (cont.) Structural analysis

connected components, contour processing, distance transform, various moments, template matching, Hough transform, polygonal approximation, line fitting, ellipse fitting, Delaunay triangulation

Camera calibration finding and tracking calibration patterns, calibration, fundamental

matrix estimation, homography estimation, stereo correspondence Motion analysis

optical flow, motion segmentation, tracking Object recognition

eigen-methods, HMM Basic GUI

display image/video, keyboard and mouse handling, scroll-bars Image labeling

line, conic, polygon, text drawing

Emgu CV Emgu CV is a cross platform .Net wrapper to the Intel

OpenCV image processing library. Allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++, IronPython etc.

Home Page: http://www.emgu.com/wiki/index.php/Main_Page

Online Documentation: http://www.emgu.com/wiki/files/2.2.1/document/Index.html

Download the executable version and install it. Version 2.1.0 is on our course website:

http://faculty.uscupstate.edu/svandelden

Getting Started… Create a new Windows Form Application

Add the Emgu Libraries as project References

Setting up the Window Open your toolbox and add a SplitContainer to your Window. Split it horizontally

This creates two “Panel”s where we can add elements. Idea: we are going to add the

video stream to the below panel and results/data etc to the top panel.

You can change the properties of any of the elements in the Windows form by clicking on the Properties to the right…

When you are building the Windows Form, you are in “Design” view… Right click on it and choose “View Code” to see the underlining code:

Notice that this is a partial class.

The InitializeComponent() method is automatically created and creates the Windows Form that you’ve been building.

To view the other partial class, click on the “designer” cs file in the Solution Explorer.

If you expand this, you can see all the code that sets up the Windows Form. NOTE: Never modify this code directly at all, because it gets re-created every time you make changes to the design of the form!!

From your Toolbox, add an ImageBox to the lower panel. Drag he Panel sizes so that it looks like this.

Note: if you are having problems selecting the SplitContainer, right click on the form and choose “Select SplitContainer”

Add a button labeled “Start Camera” to Panel1, then double click on it to create and view a button1_Click method (event handler)…

Include the EMGU libraries.

Create two data attributes :

Capture _capture is the EMGU CV class for connecting to a camera

The boolean will be used in toggling the camera off and on.

Button Click event handler.

Add a user-defined method called ProcessFrame that will be doing our computer vision stuff. For now, just have it get the camera’s current frame and add it to the ImageBox on the Windows Form.

Add this code to the button’s click method:

Instantiate the EMGU CV Camera object

Toggle the camera on and off.

Try Running the program and click the Start Camera button.

Try applying a built-in EMGU CV computer vision algorithm to the image. For example, convert to gray scale image

and threshold (cut off 100 and include pixels up to 255:

Try it out..

Try Using the Inverted Threshold method (so that the background is black) and also do an “closing” – dilation followed by erosion.

Camera/Robot Calibration

Camera Calibration

Camera/Robot Calibration We will calibrate the camera’s coordinate

system with the robot’s world coordinate system.

We assume that the depth (Z) of the points on robot world system are known and equal. Flat table top of the robot’s work area This is significant because the robot world points

form a plane in 3D space. Since depth is known, we only need to calibrate

camera (X,Y) pairs to robot (X,Y) pairs

3x3 Perspective Transformation Emgu has a built-in method that will compute

the 3x3 perspective transformation based on at least 4 camera-robot (X,Y) pairs. Need at least 4 points (8 values in total) because

we need to recover 8 values:

a b cd e fg h 1

The Transformation

The transformation captures/recovers:Translation: Scaling:

Rotation: Shear:

The Emgu CV

HomographyMatrix PerM = CameraCalibration.GetPerspectiveTransform(

PointF[ ] CameraPoints, PointF[ ] RobotPoints);

NOTE: “Homography” – square, invertible matrix.

Example Program

Provide for you is a C# calibration program that connects to the V+ robot via TCP/IP displays the video from the camera allows you to click on the image

When you do so, the picture X,Y along with the robot X,Y location of the tool frame are captured.

generate the calibration matrix based on the recovered points.

Activity

Use this program to calibrate your camera to your robot.