Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in...
Transcript of Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in...
![Page 1: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/1.jpg)
Introduction
Erkut Erdem
BIL717, February 2012!
![Page 2: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/2.jpg)
Instructor and Course Schedule
• Dr. Erkut ERDEM • [email protected] • Office: 114
• Tel: 297 7500 / 149
• Lectures: Tuesday, 13:30-16:30
• Office Hour: Friday, 13:00-15:00
![Page 3: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/3.jpg)
Communication
• The course webpage will be updated regularly throughout the semester with lecture notes, programming and reading assignments and important deadlines.
• All other communications will be carried out through Piazza. Please enroll it by following the link https://piazza.com/hacettepe.edu.tr/spring2012/bil717
![Page 4: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/4.jpg)
Prerequisites
• Programming skills ���(C/C++, Matlab)
• Good math background ���(Calculus, Linear Algebra, Statistical Methods)
• A prior, introductory-level course in image processing is recommended.
![Page 5: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/5.jpg)
Reading Material
• Lecture notes and handouts • Papers and journal articles
![Page 6: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/6.jpg)
Reference Books
• Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations, G. Aubert and P. Kornprobst, ���2nd Edition, Springer-Verlag, 2006
• Image Processing And Analysis: Variational, PDE, Wavelet, And Stochastic Methods, ���T. Chan and J. Shen, Society for Industrial and Applied Mathematics, 2005
![Page 7: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/7.jpg)
Related Conferences
• International Conference on Scale Space and Variational Methods in Computer Vision (SSVM)
• Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR)
• IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• Advances in Neural Information Processing Systems (NIPS) • IEEE International Conference on Computer Vision (ICCV) • European Conference on Computer Vision (ECCV) • IEEE International Conference on Pattern Recognition (ICPR) • IEEE International Conference on Image Processing (ICIP) • British Machine Vision Conference (BMVC)
![Page 8: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/8.jpg)
Related Journals
• IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE TPAMI)
• IEEE Transactions on Image Processing (IEEE TIP) • Journal of Mathematical Imaging and Vision (JMIV) • International Journal of Computer Vision (IJCV)
• Computer Vision and Image Understanding (CVIU) • Image and Vision Computing (IMAVIS) • Pattern Recognition (PR)
![Page 9: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/9.jpg)
Grading Policy
• Class participation, 5% • Assignments, 30% • Project, 30%
• Final Exam, 35%
![Page 10: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/10.jpg)
Assignments
• There will be at least four assignments related to the topics covered in the class.
• Each assignment will involve implementing an algorithm, carrying out a set of experiments to evaluate it, and writing up a report on the experimental results.
• There will also be some warm-up and reading assignments.
• All assignments have to be done individually, unless stated otherwise.
![Page 11: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/11.jpg)
Project
• The students taking the course will be required to do a project in computer vision.
• Students can choose to work individually or in groups of at most 2 people.
• This project may be – An original implementation of a new or published study – A detailed empirical evaluation of two or more related methods
not covered in the class
• March 20th: Brainstorming session on project previews • April 3rd: Project Proposals
![Page 12: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/12.jpg)
Tentative Outline
• (1 week) Introduction, course policy • (1 week) Review of basic concepts • (4 weeks) Image de-noising
• (4 weeks) Boundary/region detection • (2 weeks) Selected topics • (2 weeks) Project presentations
![Page 13: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/13.jpg)
Image Processing
![Page 14: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/14.jpg)
Fundamentals of Image Processing
Reality Image Formation (Software - Hardware)
Digital ���Image
Image Processing
Another���Digital Image
Information
• What is a digital image, how it is formed?
• How images are represented in computers?
• Why we process images?
• How we process images?
![Page 15: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/15.jpg)
Image Formation
Three Dimensional ���World
Two Dimensional ���Image Space
• What is measured in an image location?
– brightness
– color
viewpoint illumination conditions local geometry local material properties
<<
Figures: Francis Crick, The Astonishing Hypothesis, 1995!
![Page 16: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/16.jpg)
Image Formation
Figures: Gonzalez and Woods, Digital Image Processing, 3rd Edition, 2008!
• Discretization - in image space - sampling
- In image brightness - quantization
![Page 17: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/17.jpg)
Image Representation
• Digital image: 2D discrete function f • Pixel: Smallest element of an image f(x,y) The raster image (pixel matrix)
Figure: M. J. Black!
![Page 18: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/18.jpg)
Image Representation
• Digital image: 2D discrete function f • Pixel: Smallest element of an image f(x,y)
Figure: M. J. Black!
The raster image (pixel matrix) 0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99 0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91 0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92 0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95 0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85 0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33 0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74 0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93 0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99 0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97 0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
![Page 19: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/19.jpg)
• Visual perception as a data-driven, bottom-up process ���(traditional view since D. Marr)
• Unidirectional information flow
• Simple low-level cues >> Complex abstract perceptual units
Visual Modules and the Information Flow
![Page 20: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/20.jpg)
Visual Modules and the Information Flow
• Vision modules can be categorized into three groups ���according to their functionality: – Low-level vision: filtering out irrelevant image data
– Mid-level vision: grouping pixels or boundary fragments together
– High-level vision: complex cognitive processes
![Page 21: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/21.jpg)
Sample Image Processing Applications
• Image Filtering • Image Segmentation • Image Registration
• Image Inpainting • Seam Carving • Image Analysis
• Image Compression • …
![Page 22: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/22.jpg)
Image Filtering
• Filtering out the irrelevant information
• Image denoising, image sharpening, image smoothing, image deblurring, etc.
• Edge detection
observed���image
desired���image
irrelevant���data
![Page 23: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/23.jpg)
Edge Detection
• Uniformity of intensity or color • Texture uniformity
Canny edge detector
![Page 24: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/24.jpg)
Image Filtering
• Difficulty: Some of the irrelevant image information have characteristics similar to those of important image features
![Page 25: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/25.jpg)
Image Smoothing - A Little Bit of History
• Gaussian Filtering / linear diffusion – the most widely used method
• mid 80’s – unified formulations – methods that combine smoothing and edge detection – Geman & Geman’84, Blake & Zisserman’87, ���
Mumford & Shah’89, Perona & Malik’90
![Page 26: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/26.jpg)
Non-local Means Denoising
A. Buades, B. Coll, J. M. Morel, A non-local algorithm for image denoising, CVPR, 2005!
!!
![Page 27: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/27.jpg)
Context-Guided Filtering
• Use local image context to steer filtering
E. Erdem and S. Tari, Mumford-Shah Regularizer with Contextual Feedback, JMIV, 2009!
![Page 28: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/28.jpg)
Image Segmentation
• Partition an image into meaningful regions that are likely to correspond to objects exist in the image
Figures: A. Erdem!
![Page 29: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/29.jpg)
Image Segmentation
• Difficulty: Images may have missing or corrupting features due to partial occlusion, noise and weak edges.
![Page 30: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/30.jpg)
Image Segmentation
• Boundary-based segmentation • Region-based segmentation • Unified formulations
![Page 31: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/31.jpg)
Snakes
M. Kass, A. Witkin, and D. Terzopoulos, Snakes: Active Contour Models, IJCV, 1988!
• Curve Evolution - parametric curve formulation
![Page 32: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/32.jpg)
Active Contours Without Edges
T. Chan and L. Vese. Active Contours Without Edges, IEEE Trans. Image Processing, 2001!
• Curve Evolution – a level-set based curve formulation
![Page 33: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/33.jpg)
Normalized Cuts
• A graph-theoretic formulation
J. Shi and J. Malik, Normalized Cuts and Image Segmentation, IEEE Trans. Pattern Anal. Mach. Intel.!
![Page 34: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/34.jpg)
Prior-Shape Guided Segmentation
• Incorporate prior shape information into the segmentation process
Our result Deformation map E. Erdem, S. Tari, and L. Vese, Segmentation Using The Edge Strength Function as a Shape Prior
within a Local Deformation Model, ICIP 2009!
![Page 35: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/35.jpg)
Registration
• Estimate a transformation function between – two images
– two point sets – two shapes
– …
![Page 36: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/36.jpg)
Registration
H. Chui and A. Rangarajan, A new point matching algorithm for non-rigid registration, CVIU, 2003!
![Page 37: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/37.jpg)
Image Registration
(top) Alain Trouve and Laurent Younes, Metamorphoses Through Lie Group Action, Found. Comput. Math., 2005 (bottom) M. I. Miller and L. Younes, Group Actions, Homeomorphisms, and Matching: A General Framework, IJCV, 2001!
![Page 38: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/38.jpg)
Image Inpainting
• Reconstructing lost or deteriorated parts of images
M. Bertalmio, G. Sapiro, V. Caselles and C. Ballester, Image Inpainting, SIGGRAPH, 2000!
![Page 39: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/39.jpg)
Seam Carving
• Content-aware image resizing
S. Avidan and A. Shamir, Seam Carving for Content-Aware Image Resizing, SIGGRAPH, 2007!
![Page 40: Introduction - Hacettepe Üniversitesierkut/bil717.s12/w01-intr.pdf · • Mathematical Problems in Image Processing: Partial Differential Equations and the Calculus of Variations,](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ed32051e9c55e2ba56a6f1b/html5/thumbnails/40.jpg)
Image Analysis
• Extracting information from images
• Image/Scene Classification
• Object Detection/Recognition • Shape Analysis