Information from pixels
-
Upload
dave-snowdon -
Category
Technology
-
view
102 -
download
0
Transcript of Information from pixels
Information from PixelsDave Snowdon@davesnowdon
https://github.com/davesnowdon/ljc-information-from-pixelshttp://www.slideshare.net/DaveSnowdon1/information-from-pixels
Summary• Why? What?• Range operations and colour spaces• Kernels & convolution• Object detection• Contours• Conclusion
Why me?
• Social robotics developer• Social robots need to handle unstructured
environments• Vision is the most versatile way of sensing
the environment
Machine vision• Tracking movement: Dyson 360, Google
Tango• Recognising people, biometric security• Recognising medication• Image search• …
Why this is hard
https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/
Why this is hard• Colour reproduction, lighting & white
balance• Perspective & rotation effects• Noise• Different scales
The good news• Open source• Tried and tested• Large collections of algorithms• Language bindings for C, python & java• Runs on pretty much anything (Linux, Mac,
Windows, android, iOS, RaspberryPi)
L*a*b* / CIELAB
https://gurus.pyimagesearch.com/wp-content/uploads/2015/03/color_spaces_lab_axis.jpg
Get an image• From a Java image• From video / webcam
org.opencv.videoio.VideoCapture• From fileimport org.opencv.core.Mat;Mat image = Imgcodecs.imread(filename);
org.opencv.core.Mat
new Mat(numRows, numColumns, CvType.CV_8UC3);
• Dense multi-dimensional matrix• Variants with int, double, byte values• Implements basic matrix operations
B G R B G R B G R B G RB G R B G R B G R B G RB G R B G R B G R B G R
Erode & Dilatefinal Mat se = Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, new Size(kernelSize, kernelSize));
Imgproc.erode(image, result, se, new Point(-1, -1), numIterations);
Imgproc.dilate(image, result, se, new Point(-1, -1), numIterations);
Find contours
Imgproc.findContours(image, contours, new Mat(), Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
Find largest contourcontours.stream() .max((c1, c2) -> (Imgproc.contourArea(c1) > Imgproc.contourArea(c2) ? 1 : -1)) .get();
Draw contour (for demo)
Imgproc.circle(image, centre, 5, CENTRE_COLOUR, 2);
Imgproc.drawContours(image, Arrays.asList(contour), 0, OUTLINE_COLOUR, 2);
Output image• Don’t always need to• Grab region of interestMat roi = mat.submat(Rect)• Convert to java imageBufferedImage javaImage = Util.matrixToImage(mat);Util.displayImage(command, javaImage);
• Write to fileImgcodecs.imwrite(filename, mat);
Built-in blob detection
• OpenCV has built-in blob detection: SimpleBlobDetector• blob detection by colour may not work
• Blog post: https://www.learnopencv.com/blob-detection-using-opencv-python-c/
Convolution
https://developer.apple.com/library/mac/documentation/Performance/Conceptual/vImage/ConvolutionOperations/ConvolutionOperations.html
Detecting blurred images
• Want to discard images that are unlikely to be of use
• The more blurred an image is the fewer sharp edges will be found
• What happens to the laplacian of an image as it’s blurred…
Code// apply laplacian to grayscale copy of imageImgproc.Laplacian(gray, laplacian, CvType.CV_64F);
// determine varianceMatOfDouble mean = new MatOfDouble();MatOfDouble stddev = new MatOfDouble();Core.meanStdDev(laplacian, mean, stddev);double sd = stddev.toList().get(0);double var = sd * sd;
Kernel to detect vertical lines
-1 2 -1
Mat kernel = new Mat(1, 3, CvType.CV_64F);double[] kernel_values = {-1.0, 2.0, -1.0};kernel.put(0, 0, kernel_values);
Sliding window
http://www.pyimagesearch.com/2015/03/23/sliding-windows-for-object-detection-with-python-and-opencv/
Boosting• Train all features on every training
example• For each feature find the best threshold
which distinguished positive from negative• Select features with minimum error rate• Final classifier is weighted sum of these
weak classifiers
Cascade• Hugely expensive to compute all features on
every window location• Group features into different stages with
smaller number of features• Only proceed to next stage when previous
stage passes• In Viola-Jones paper as few as 10 features
out of 6000 might be evaluated per window
Pre-trained classifiers• front face• profile face• Full body• Upper body• Lower body• Left & right eyes (one classifier each for left & right)• Smile• Front cat face• Russian license plate
Using a classifier// create classifier object from XML definitionfinal CascadeClassifier faceClassifier = new CascadeClassifier(classifierFilename);
// apply classifer to get list of matching regionsfinal MatOfRect mor = new MatOfRect();clr.detectMultiScale(image, mor);List<Rect> result = mor.toList();
How to train• Create sample vectors from text files listing +ve & -ve images• opencv_createsamples -info positives.txt -num 68 -w 60 -h 98 -
vec nao.vec• Train• Haar: opencv_traincascade -data classifier -vec samples.vec -bg
negatives.txt -numStages 20 -minHitRate 0.999 -maxFalseAlarmRate 0.5 -numPos 1000 -numNeg 600 -w 60 -h 98 -mode ALL -precalcValBufSize 1024 -precalcIdxBufSize 1024
• LBP : opencv_traincascade -data classifier.lbp -vec samples.vec -bg negatives.txt -numStages 20 -minHitRate 0.999 -maxFalseAlarmRate 0.5 -numPos 1000 -numNeg 600 -w 60 -h 98 -featureType LBP -precalcValBufSize 1024 -precalcIdxBufSize 1024
Training docs & tutorials
• http://docs.opencv.org/trunk/dc/d88/tutorial_traincascade.html
• http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html
Find contours// use Canny edge detector on blurred grayscale imageImgproc.Canny(blurred, edges, 75, 200);
// find contoursImgproc.findContours(image, contours, new Mat(), Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
Type conversion
// need to convert the contour from a MatOfPoint to MatOfPoint2ffinal MatOfPoint2f m2f = new MatOfPoint2f();m2f.fromList(contour.toList());
Approximate shapes// approximate contour polygon with 1% or less difference in perimeterdouble perimeter = Imgproc.arcLength(m2f, true);MatOfPoint2f approx = new MatOfPoint2f();Imgproc.approxPolyDP(m2f, approx, 0.01 * perimeter, true);
// check number of line segmentsint numSides = approx.toList().size();
More information• OpenCV docs: http://docs.opencv.org/3.1.0/• Useful blogs:
• http://www.pyimagesearch.com • https://www.learnopencv.com
• https://opencv-java-tutorials.readthedocs.io/en/latest/
• Code for examples: https://github.com/davesnowdon/ljc-information-from-pixels