C#, OpenCV, and simple Camera/Robot Calibration
description
Transcript of C#, OpenCV, and simple Camera/Robot Calibration
![Page 2: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/2.jpg)
OpenCV
Open Source Computer Vision Library OpenCV is Intel’s Open Source Computer
Vision Library. It is a collection of C functions and a few C++
classes that implement some popular Image Processing and Computer Vision algorithms.
EMGU provides wrapper class packages so that OpenCV can be used in C#.
![Page 3: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/3.jpg)
3
Features Image data manipulation
allocation, release, copying, setting, conversion Image and video I/O
file and camera based input, image/video file output Matrix and vector manipulation, and linear algebra
routines products, solvers, eigenvalues, SVD
Various dynamic data structures lists, queues, sets, trees, graphs
Basic image processing filtering, edge detection, corner detection, sampling and
interpolation, color conversion, morphological operations, histograms, image pyramids
![Page 4: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/4.jpg)
4
Features (cont.) Structural analysis
connected components, contour processing, distance transform, various moments, template matching, Hough transform, polygonal approximation, line fitting, ellipse fitting, Delaunay triangulation
Camera calibration finding and tracking calibration patterns, calibration, fundamental
matrix estimation, homography estimation, stereo correspondence Motion analysis
optical flow, motion segmentation, tracking Object recognition
eigen-methods, HMM Basic GUI
display image/video, keyboard and mouse handling, scroll-bars Image labeling
line, conic, polygon, text drawing
![Page 5: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/5.jpg)
Emgu CV Emgu CV is a cross platform .Net wrapper to the Intel
OpenCV image processing library. Allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++, IronPython etc.
Home Page: http://www.emgu.com/wiki/index.php/Main_Page
Online Documentation: http://www.emgu.com/wiki/files/2.2.1/document/Index.html
Download the executable version and install it. Version 2.1.0 is on our course website:
http://faculty.uscupstate.edu/svandelden
![Page 6: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/6.jpg)
Getting Started… Create a new Windows Form Application
![Page 7: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/7.jpg)
Add the Emgu Libraries as project References
![Page 8: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/8.jpg)
Setting up the Window Open your toolbox and add a SplitContainer to your Window. Split it horizontally
This creates two “Panel”s where we can add elements. Idea: we are going to add the
video stream to the below panel and results/data etc to the top panel.
![Page 9: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/9.jpg)
You can change the properties of any of the elements in the Windows form by clicking on the Properties to the right…
![Page 10: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/10.jpg)
When you are building the Windows Form, you are in “Design” view… Right click on it and choose “View Code” to see the underlining code:
Notice that this is a partial class.
The InitializeComponent() method is automatically created and creates the Windows Form that you’ve been building.
![Page 11: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/11.jpg)
To view the other partial class, click on the “designer” cs file in the Solution Explorer.
If you expand this, you can see all the code that sets up the Windows Form. NOTE: Never modify this code directly at all, because it gets re-created every time you make changes to the design of the form!!
![Page 12: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/12.jpg)
From your Toolbox, add an ImageBox to the lower panel. Drag he Panel sizes so that it looks like this.
Note: if you are having problems selecting the SplitContainer, right click on the form and choose “Select SplitContainer”
![Page 13: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/13.jpg)
Add a button labeled “Start Camera” to Panel1, then double click on it to create and view a button1_Click method (event handler)…
Include the EMGU libraries.
Create two data attributes :
Capture _capture is the EMGU CV class for connecting to a camera
The boolean will be used in toggling the camera off and on.
Button Click event handler.
![Page 14: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/14.jpg)
Add a user-defined method called ProcessFrame that will be doing our computer vision stuff. For now, just have it get the camera’s current frame and add it to the ImageBox on the Windows Form.
![Page 15: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/15.jpg)
Add this code to the button’s click method:
Instantiate the EMGU CV Camera object
Toggle the camera on and off.
![Page 16: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/16.jpg)
Try Running the program and click the Start Camera button.
![Page 17: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/17.jpg)
Try applying a built-in EMGU CV computer vision algorithm to the image. For example, convert to gray scale image
and threshold (cut off 100 and include pixels up to 255:
![Page 18: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/18.jpg)
Try it out..
![Page 19: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/19.jpg)
Try Using the Inverted Threshold method (so that the background is black) and also do an “closing” – dilation followed by erosion.
![Page 20: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/20.jpg)
Camera/Robot Calibration
![Page 21: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/21.jpg)
Camera Calibration
![Page 22: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/22.jpg)
Camera/Robot Calibration We will calibrate the camera’s coordinate
system with the robot’s world coordinate system.
We assume that the depth (Z) of the points on robot world system are known and equal. Flat table top of the robot’s work area This is significant because the robot world points
form a plane in 3D space. Since depth is known, we only need to calibrate
camera (X,Y) pairs to robot (X,Y) pairs
![Page 23: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/23.jpg)
3x3 Perspective Transformation Emgu has a built-in method that will compute
the 3x3 perspective transformation based on at least 4 camera-robot (X,Y) pairs. Need at least 4 points (8 values in total) because
we need to recover 8 values:
a b cd e fg h 1
![Page 24: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/24.jpg)
The Transformation
The transformation captures/recovers:Translation: Scaling:
Rotation: Shear:
![Page 25: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/25.jpg)
The Emgu CV
HomographyMatrix PerM = CameraCalibration.GetPerspectiveTransform(
PointF[ ] CameraPoints, PointF[ ] RobotPoints);
NOTE: “Homography” – square, invertible matrix.
![Page 26: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/26.jpg)
Example Program
Provide for you is a C# calibration program that connects to the V+ robot via TCP/IP displays the video from the camera allows you to click on the image
When you do so, the picture X,Y along with the robot X,Y location of the tool frame are captured.
generate the calibration matrix based on the recovered points.
![Page 27: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/27.jpg)
![Page 28: C#, OpenCV, and simple Camera/Robot Calibration](https://reader033.fdocuments.us/reader033/viewer/2022061314/56815cb7550346895dcab364/html5/thumbnails/28.jpg)
Activity
Use this program to calibrate your camera to your robot.