Post on 26-Dec-2015
ENEE631 Digital Image Processing (Spring'09)
Image Forensics & Feature ExtractionImage Forensics & Feature Extraction
Spring ’09 Instructor: Min Wu
Electrical and Computer Engineering Department,
University of Maryland, College Park
bb.eng.umd.edu (select ENEE631 S’09) minwu@eng.umd.edu
ENEE631 Spring’09ENEE631 Spring’09Lecture 23 (4/27/2009)Lecture 23 (4/27/2009)
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
04
)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [2]
Overview and LogisticsOverview and Logistics
Last Time:
– Robust watermark via spread spectrum embedding– Two major types and unification of embedding strategies– Digital forensic fingerprinting for traitor tracing
Today:
– Non-intrusive image forensics– Useful image features and feature extraction techniques
Gradient based; Projection based
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
04
)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [7]
Recall: Issues to Address in Data HidingRecall: Issues to Address in Data Hiding
Tradeoff among conflicting requirements– Imperceptibility– Robustness & security– Capacity
Key elements of data hiding– Perceptual model– Embedding one bit– Multiple bits– Uneven embedding capacity– Robustness and security– What data to embed
Up
per
L
ayer
s
Uneven capacity equalization
Error correction
Security
……
Low
er
Lay
ers
Imperceptible embeddingof one bit
Multiple-bit embedding
Coding of embedded data
Robustness
Capacity
Imperceptibility
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
ba
sed
on
Re
sea
rch
Ta
lks
’98
-’04
)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [8]
Embedded Fingerprint for Tracing TraitorsEmbedded Fingerprint for Tracing Traitors
Insert special signals to identify recipients
– Deter leak of proprietary documents– Complementary protection to encryption– Consider imperceptibility, robustness, traceability
– Attacks mounted by single and multiple users
Collusion Attack Collusion Attack (to remove fingerprints)(to remove fingerprints)
AliceAlice
BobBob
Colluded CopyColluded Copy
Unauthorized Unauthorized rere--distributiondistribution
Fingerprinted docfor different users
Collusion Attack Collusion Attack (to remove fingerprints)(to remove fingerprints)
AliceAlice
BobBob
Colluded CopyColluded Copy
Unauthorized Unauthorized rere--distributiondistribution
Fingerprinted docfor different users
Extract Extract FingerprintsFingerprints
Suspicious Suspicious CopyCopy
101110 …101110 …
Codebook
Alice, Bob, …
Identify Identify TraitorsTraitors
Extract Extract FingerprintsFingerprints
Suspicious Suspicious CopyCopy
101110 …101110 …
Codebook
Alice, Bob, …
Identify Identify TraitorsTraitors
Multi-user Attacks
Traitor Tracing
President
Satellite Image
Alice
Bob
Carl
w1
w2
w3
LeakLeak
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [11]
Fingerprinting Topographic MapFingerprinting Topographic Map
– Traditional protection: intentionally alter geospatial content
– Embed much less intrusive digital fingerprint for a modern protection
• 9 long curves are marked; 1331 control points used to carry the fingerprint
1100x1100 Original Map Fingerprinted Map
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [12]
Collusion-Resistant Fingerprinting of MapsCollusion-Resistant Fingerprinting of Maps
2-User Interleaving Attack5-User Averaging Attack
. . .
Can survive combined attacks of collusion + print + scan
Can extend to 3-D Digital Elevation Map
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [14]
Many Forensic Questions …Many Forensic Questions …
arise from military, intelligence, law enforcement, and commercial applications
What type of sensor was used?
Which camera brand took this picture? What model?
What processing has been done?
– Has it been tampered? manipulated?
What technologies were employed?
– Given two images, are they acquired by devices with similar imaging technologies?
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [15]
Break down the info. processing chain into individual components
Identify algorithms and parameters employed in major components of a digital device or processing system
Exploit Intrinsic Fingerprints via Component ForensicsExploit Intrinsic Fingerprints via Component Forensics
Ref: Swaminathan/Wu/Liu in ICASSP’06 and IEEE Trans. Info Forensics & Security (’07)
Color Filter Array (CFA)
ColorInterpolation
White Balancing
Real world scene Digital imageCamera Components
… Sensors …
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [19]
Forensic Estimation and IdentificationForensic Estimation and Identification
Establish a processing model and estimate parameters– Small # possibilities => exhaustive search or by classifier design – More continuous valued parameters => analyze w/ estimation theory
Example: color interpolation in digital camera– Approximate by texture classification and linear filter
(one set of interpolation coeff. for smooth, horizontal & vertical)
– Find best linear estimate of filter coeff. in each class(least-square type of method for robustness)
– Find CFA pattern in a search space that minimizes fitting errors
CFA Interpolation
R ?
? ?
R ?
? ?
R ?
? ?
R ?
? ?
? ?? ?
CandidateCFA pattern
A x = b Interp. equation set
b ~ interpolated pixels; A ~ each row is for neigh-borhood of a interp. pixel (based on directly sensed, known pixels)
Interp. coeff. x and fitting error for each region type and color
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [20]
Experiments with Images from Digital CamerasExperiments with Images from Digital Cameras
Camera Model Camera Model
123456789
10
Canon Powershot A75Canon Powershot S400Canon Powershot S410Canon Powershot S1 IS Canon Powershot G6Canon EOS Digital REBELNikon E4300 Nikon E5400Sony Cybershot DSC P7Sony Cybershot DSC P72
111213141516171819
Olympus C3100Z/C3020ZOlympus C765UZMinolta DiMage S304Minolta DiMage F100Casio QV-UX2000FujiFilm Finepix S3000FujiFilm Finepix A500Kodak CX6330Epson PhotoPC 650
19 cameras and 200 image blocks per camera model
– 512 x 512 regions with maximum gradients chosen for analysis (s.t. have substantial revealing evidence on color interpolation)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [21]
Detecting Which Camera Detecting Which Camera BrandBrand Took the Image Took the Image
Canon Nikon Sony Olympus Minolta Casio Fuji Kodak Epson
Canon 96% * * * * * * * *
Nikon * 83% 5% * * * * * *
Sony * * 90% * * * * * *
Olympus * * * 93% * * * * *
Minolta 8% * * * 81% * * * *
Casio * * * 6% * 89% * * *
Fuji * * * * 7% * 87% * *
Kodak * * * * * * * 89% *
Epson * * * * * * * * 100%
Interpolation coefficients as features for classifier Average accuracy: 90% for 9 camera brands on uncontrolled scenes Best related work under controlled, uncompressed setting on input scenes
84% for 3 brands, uncompressed [Kharrazi et al’ 05]; 96% for 3 brands [Bayram et al’ 06]
(* denotes values below 4%)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [23]
Tampering DetectionTampering Detection
Explore intrinsic fingerprints left by various processing modules
– To infer the algorithms and parameters employed in various components of the digital device and processing systems
– New traces or vanished old traces suggests potential post-camera operations
Estimated coeff.
From direct camera output
After post-camera filtering
ColorInterpolation
ColorSensors
Scene Optical Lens System CAMERA
Other SoftwareProcessing
ATampering
/ Stego
B
Black: Sony P72; White: Canon Powershot S410
Grey: Classified as other cameras with low confidence
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [24]
Overview of Scanner ModelOverview of Scanner Model
Tri-linear CCD sensors
Tri-linear CCD sensorsHardcopy
graphic data
Lamp & MirrorsLamp & Mirrors
Lens
Tri-linearcolor filter array
Scanner head
Motion systemMotion system
Shift RegisterAmplifier
A/D Converter
Shift RegisterAmplifier
A/D Converter
Post-processing
Interpolation, Color transformation White balancing, Exposure controlNoise reduction,…
Digitalimage
Software operation
Scanning noise
StatisticalStatisticalnoise featuresnoise features
Scanner modelIdentification
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [26]
E.g.: Noise Features from Wavelet AnalysisE.g.: Noise Features from Wavelet Analysis
ScannedImage I
One stagewavelet
decomposition
HHHLLH
Subband STD &goodness of
Gaussian fitting
Statisticalfeatures
f (3)(I), f (4)(I)
Digital photograph Scanner model 1 Scanner model 2
Histogram Mean and STD Gaussian distribution Goodness of Gaussian fitting
Fitting errorFitting error
HH,HL,LH sub-bandsRGB components2x3x3 = 18 features
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [28]
Acquisition Forensics w/ Noise + Interp. FeaturesAcquisition Forensics w/ Noise + Interp. Features
Q1) What type of device was used to capture the image?
94% accuracy in identifying device type
Q2) What brand/model of the device captured the image?
Cellphone cameras: 98% accuracy over 5 brands
Standalone cameras: 90% over 19 camera models from 9 camera brands
Scanners: 93% accuracy over 9 scanner brands
Cell Phone Camera
Standalone Camera
Scanner
Computer Generated
Input Image
Brand/Model
IdentificationAcquisition Device
Type Identification
Further Forensic Analysis
Sony
Samsung
Nokia
AudiovoxMotorola
CanonFujiFilm
CasioMinolta
Epson
Microtek
AcerScan
Canon
Step 1Step 2
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [30]
Useful Image Features Useful Image Features
and Feature Extraction Techniquesand Feature Extraction Techniques
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [31]
Types of Image Processing TasksTypes of Image Processing Tasks
Image in, Image out– Codec (compression-decompression)– Image enhancement and restoration– Digital watermarking
=> May require both analysis and synthesis operations Intermediate output may be non-image like (coded stream),
but end output should reconstruct into an image close/relate to the input
Image in, Features out
– Features may be used for classification, recognition, and other studies
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [32]
Line FeaturesLine Features
Profile – Project image intensity or other
properties along a given direction– Useful for document and object
analysis
Example-1 locate text linesExample-2 reading banker font
1-D signal via narrow reading head Equiv. to vertical projection,
then 1st order derivative.
Identify colinear points and linesvia Hough transform
Figure 12.7 from Gonzalez’s book resource: “American Bankers Association E-13B font character set and corresponding waveforms
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [33]
Hough TransformHough Transform
(x, y) space parameter space (a, b) y = a x + b b = x a + y
(, ) space: x cos + y sin =
– Representing in (, ) space has similarity to Radon transform
Ref: P. V. C. Hough, "Method and means for recognizing complex patterns." U.S. Patent 3,069,654, 1962.
Figures from Gonzalez’s book resource for Chapter 10
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [34]
– For each point (x0, y0), lines of all angles passing it form a sinusoid curve in (, ) space
– (, ) curves corresponding to colinear points intersect at a point (0, 0)
=> useful for line detection
Illustration of Illustration of Hough TransformHough Transform
Figures from http://en.wikipedia.org/wiki/Hough_transform
(, ) space: x cos + y sin =
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [35]
Example: Detecting Lines in Hough Transf. DomainExample: Detecting Lines in Hough Transf. Domain
Intensity peaks of the transformed results correspond to the (, ) of the respective lines
Can extend to other patterns such as circles
Figures from http://en.wikipedia.org/wiki/Hough_transform
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [36]
Corner DetectorCorner Detector
Corner: intersection of 2 edges– Large variation in local intensity in multiple directions– Alleviate “aperture problem” in motion analysis & image matching
Harris-Stephens detector (1988)– Examine (weighted) Sum of Squared Difference
between an image patch& neighborhood
– Build a 2x2 Harris matrix A Describes 2nd order derivatives of the above SSD
averaged over neighborhood
– Examine the two eigen values of A If both close to 0, there are no features of interest at pixel (x,y). If one close to 0 and the other has large positive value, then an edge is
found. If both have large, distinct positive values, then a corner is found. May avoid computing eigen: examine M = det(A) – k tr(A)2 with k =
0.04-0.15
Ref. and examples at http://en.wikipedia.org/wiki/Corner_detection
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [38]
More on Gradient-based Salient Feature PointsMore on Gradient-based Salient Feature Points Recall operations for robust edge detection ~ e.g. Canny
– Laplacian of Gaussian (LoG) [Gonzalez 3/e 10.2.6] effectively bandpass filtering to suppress noise when taking
derivatives
– Ridge of gradient magnitude and edge linking Local extrema of gradient; seek stable edge info
Recall multiresolution analysis
Achieve invariance to scaling and rotation
– Take account of multiple resolution scale levels– Represent orientation w.r.t. dominant or canonical direction
=> Scale Invariant Feature Transform (SIFT) by D. Lowe Give about 2000 stable “keypoints” for a typical 500 x 500 image Each keypoint is described by a vector of 4 x 4 x 8 = 128 elements
(over 4x4 array of 8-bin gradient histograms in keypoint neighborhood)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [39]
Recall: Robust Edge DetectorRecall: Robust Edge Detector Apply LPF to suppress noise, then apply edge detector or
derivative operations
E.g. Laplacian of Gaussian (LoG): in shape of Mexican hat
Figures from Gonzalez-Woods 2/e online slides.
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [41]
SIFT: Employ “Scale Space Extrema”SIFT: Employ “Scale Space Extrema” Examine Differences of Gaussian filtered
images at nearby scale and k Avoid low contrast & poorly defined DoG peaks
Figures from Lowe’s IJCV 2004
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [42]
Examples of Difference of GaussianExamples of Difference of Gaussian
Examples from SIFT tutorial notes by Estrada/Jepson/Fleet (2004)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [43]
Examples from SIFT tutorial notes by Estrada/Jepson/Fleet (2004)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [44]
(weighted) Gradient orientation histogram near a stable DoG extrema– Peaks in histogram correspond to dominant orientation– Also include other directions close to the peak as keypoints for higher stability– Measure properties of a keypoint relative to its assigned orientation to gain
rotational invariance
Keypoint Descriptor records gradient pattern of each keypoint
– As a set of 8-bin orientation histograms over 4x4 nearby blocks
SIFT: Rotational Invariant Keypoint Descriptor SIFT: Rotational Invariant Keypoint Descriptor
Figure from Lowe’s IJCV 2004
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [45]
Summary of Today’s LectureSummary of Today’s Lecture
Readings
– Image Forensics: IEEE Signal Proc. Magazine March 2009
– Line and corner detection: Gonzalez’s 3/e book 10.2.7
– Stable Gradient Features ~ SIFTDavid G. Lowe (2004). "Distinctive Image Features from Scale-Invariant Keypoints". International Journal of Computer Vision 60 (2): 91–110.
For more explorations
– Boundary representation: Gonzalez’s 3/e book 11.1 – 11.2
– Morphological operation: Gonzalez’s book Chapter 9
UM
CP
EN
EE
63
1 S
lide
s (c
rea
ted
by
M.W
u ©
20
04
)
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [46]
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [47]
Pattern MatchingPattern Matching Minimum distance classifier: compare with mean feature vector of
each class (if known or can be learned)
Matching by correlation– Reduce sensitivity via normalized correlation (correlation coeff.)– Exhaustive search for size & orientations ~ computationally expensive
– Learn more on pattern classification Overview: Chapter 12 of Gonzalez’s book ENEE731 Statistical Pattern Recognition; CS Machine Learning course
Figure from Gonzalez’s 2/e book resource
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [48]
Boundary Boundary RepresentationRepresentation
Chain code
4-way or 8-way connectivity
Features/measurement obtained from object’s chain code
– Perimeter; Area; Centroid
Figure from Gonzalez’s book resource
ENEE631 Digital Image Processing (Spring'09)Lec 23 – Image Forensics & Feature
Extraction [49]
Boundary DescriptorsBoundary Descriptors
Simple descriptors
– boundary length, diameter– curvature (rate of change of slope/tangent); B-splines
Shape number
– The 1st difference of smallest magnitude of chain code to normalize chain code’s start point and orientation
Fourier descriptor
– Record point coordinates w/ a sequence of complex #– DFT coeff. of the complex sequence as Fourier descriptors
Statistical moments
– mean, variance, and higher-order moments of boundary segments and/or 1-D representation of boundary e.g. r()