“Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid...
Transcript of “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid...
![Page 1: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/1.jpg)
“Advanced” Vision
January 10, 2005
![Page 2: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/2.jpg)
Agenda
� Hodge Podge of Vision Stuff� Stereo Vision � Rigid body motion � Edge Detection �Optical Flow � EM Algorithm to locate objects
� May not be directly applicable, but we’ve tried to make it relevant.
![Page 3: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/3.jpg)
Stereo Vision
�We can judge distance based on the how much the object’s position changes.
Left Image
Left Eye Right Eye Right Image
![Page 4: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/4.jpg)
Stereo Vision
�Use the image to find the angle to the object, then apply some trig:
Left Image
angle-side-angle gives Right Image you a unique triangle
![Page 5: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/5.jpg)
Stereo Vision
�What’s the angle? X �Perspective projection
equation tells us Zxx/f = X/Z
� f is focal length, x is fpixel location
center of projection� tan(f ) = X/Z = x/f
![Page 6: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/6.jpg)
Stereo Vision
�But in a complex image, objects may be hard to identify…
�Try to match regions instead (block correlation)
![Page 7: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/7.jpg)
Stereo Vision
�Difference metric = Sum of (Li – Ri)^2
�Search horizontally for best match (least difference) 1 61
6 55 5 56
5 75 6 55 5 56
![Page 8: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/8.jpg)
Stereo Vision
�Still have a problem: unless the object is really close, the change might be small…
Left Image
Left Eye Right Eye Right Image
![Page 9: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/9.jpg)
Stereo Vision
� And many regions will be the same in both pictures, even if the object has
Left Imagemoved. � We need to apply
stereo only to “interesting” regions.
Right Image
![Page 10: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/10.jpg)
Stereo Vision
�Uniform regions are not interesting
�Patterned regions are interesting
�Let the “interest” operator be the lowest eigenvalue of a matrix passed over the region.
5 45 5 55 5 55
lowest eigenvalue = 0
5 45 1 55 5 28
lowest eigenvalue = 2.5
![Page 11: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/11.jpg)
Stereo Vision
![Page 12: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/12.jpg)
Stereo Vision
�For Maslab, the problem is simpler… can easily identify objects and compute horizontal disparity.
�To convert disparity to distance, calibrate the trig.
�Use two cameras… or mount a camera on a movable platform… or move your robot
![Page 13: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/13.jpg)
Rigid Body Motion
�Going from data association to motion�Given � a starting x1,y1,?1 � a set of objects visible in both images
�What is x2, y2, and ?2?
position one position two
![Page 14: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/14.jpg)
Rigid Body Motion
� If we only know angles, the problem is quite hard:
1
12 2 3
3
�Assume distances to objects are known.
![Page 15: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/15.jpg)
Rigid Body Motion
� If angles and distances are known, we can construct triangles:
distance between objects should be the same from both positions
![Page 16: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/16.jpg)
Rigid Body Motion
�Apply the math for a rotation: x1i = cos(?)*x2i + sin(?)*y2i + x0
y1i = cos(?)*y2i - sin(?)*x2i + y0
�Solve for x0, y0, and ? with least squares:S (x1i - cos(?)*x2i - sin(?)*y2i - x0)^2 +
(y1i - cos(?) *y2i + sin(?)*x2i - y0 )^2 �Need at least two objects to solve
![Page 17: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/17.jpg)
Rigid Body Motion
�Advantages � Relies on the world, not on odometry � Can use many or few associations
�Disadvantage � Can take time to compute
![Page 18: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/18.jpg)
Edge Detection
�Edges are places of large change �Scan the image with little computational
molecules or a ‘kernel’
0 0
1
1
-1
-1
![Page 19: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/19.jpg)
300
Edge Detection
50
100
150
200
200
50 100 150 200 250
50
100
150
50 100 150 200 250 300
![Page 20: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/20.jpg)
Edge Detection
�More sophisticated filters work better (Laplacian of Gaussian, for example)
50
100
150
200
50 100 150 200 250 300
![Page 21: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/21.jpg)
Edge Detection
�Need to choose a good value for threshold� Too small—gets lots of noise, fat edges � Too big—lose sections of edge
�What do you do with an edge? � Extract lines for a map? � Use to separate regions?
![Page 22: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/22.jpg)
Optical Flow
�Look at changes between successive images � identify moving objects� identify robot motion (flow will radiate out
from direction of motion)
�For each point on image, set total derivative of brightness change to zero: � 0 = u*Ex + v*Ey + Et
![Page 23: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/23.jpg)
Optical flow
![Page 24: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/24.jpg)
Optical Flow
�Computationally expensive and requires very fast frame rates… or very slow robots
� Idea from optical flow: looking at change between frames can help segment an image (only edges will move).
![Page 25: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/25.jpg)
EM Algorithm
�Given an image with k objects �How can we find their locations?
![Page 26: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/26.jpg)
EM Algorithm
�Assume there are k red objects �Randomly choose object locations xk, yk
�Loop: � Assign each pixel to nearest xk, yk
� Recenter xk, yk at center of all pixels associated with it
![Page 27: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/27.jpg)
EM Algorithm
�Key question: what is k? � Need to know how many objects
�Convergence criteria for random values?� Pick good guesses for centers
![Page 28: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/28.jpg)
Performance Note
� Faster access:� bufferedImage =
ImageUtil.convertImage(bufferedImage, BufferedImage.INT_RGB);
� DataBufferInt intBuffer = (DataBufferInt) bufferedImage.getRaster().getDataBuffer();
� int[] b = dataBufferInt.getData();
� Need to keep track of where pixels are: � offset = (y*width + x) � (b[offset] >> 16) & 0xFF = red or hue � (b[offset] >> 8) & 0xFF = green or saturation
� b[offset] & 0xFF = blue or value
![Page 29: “Advanced” Vision - MIT OpenCourseWare...Agenda Hodge Podge of Vision Stuff Stereo Vision Rigid body motion Edge Detection Optical Flow EM Algorithm to locate objects May not be](https://reader035.fdocuments.us/reader035/viewer/2022071506/6126cf3a6c9bf43b3b4f4665/html5/thumbnails/29.jpg)
Reminders
�No lecture tomorrow �Design Review Wednesday�Check Point Two: Friday
� If you haven’t completed check point one, you finish it today!