A Model for Image Splicing - Columbia University
Transcript of A Model for Image Splicing - Columbia University
A Model for Image Splicing
Tian-Tsong Ng, Shih-Fu ChangDepartment of Electrical EngineeringColumbia University, New York, USA
OutlineReview
Problem and MotivationOur ApproachDefinition: BicoherenceWhy Bicoherence good for splicing detection? Previous Hypothesis
Bicoherence FeaturesMagnitude featurePhase feature
Proposed Image Splicing ModelBipolar Perturbation HypothesisBicoherence of bipolar signalBipolar perturbation effect on magnitude featureBipolar perturbation effect on phase feature
Problem & Motivation: How much can we trust digital images?
General problem: Image Forgery DetectionImage Forgery: Images with manipulated or fake content(In)Famous examples:
March 2003: A Iraq war news photograph on LA Times front page was found to be a photomontageFeb 2004: A photomontage showing John Kerry and Jane Fonda together was circulated on the Internet
Adobe Photoshop: 5 million registered users
Definitions: Photomontage and Spliced Image
Specific problem: Image Splicing Detection
Photomontage: A paste-up produced by sticking together photographic images, possibly followed by post-processing (e.g. edge softening and adding noise).Spliced Image (see figures): Splicing of image fragments without post-processing. A simplest form of photomontage.
Why interested in detecting image splicing?Image splicing is a basic and essential operation in the creation of photomontage
Therefore, a comprehensive solution for photomontage detection includes detection of post-processing operations and intelligent techniques for detecting internal scene inconsistencies
spliced
spliced
Image Forgery Detection ApproachesPassive and blind approach:
Without any prior information (e.g. digital watermark or authentication signature), verifying whether an image is authentic or fake.Advantages: No need for watermark embedding or signature generation at the source side
Active approach: Fragile/Semi Fragile Digital Watermarking: Inserting digital watermark at the source side and verifying the mark integrity at the detection side.Authentication Signature:Extracting image features for generating authentication signature at the source side and verifying the image integrity by signature comparison at the receiver side.Effective when there is
A secure trustworthy cameraA secure digital watermarking algorithmA widely accepted watermarking standard
What are the qualities of authentic images?
Image AuthenticityNatural-imaging Quality
Entailed by natural imaging process with real imaging devices, e.g. camera and scannerEffects from optical low-pass, sensor noise, lens distortion, demosicking, nonlinear transformation.
Natural-scene QualityEntailed by physical light transport in 3D real-world scene with real-world objectsResults are real-looking texture, right shadow, right perspective and shading, etc.
Examples:Computer graphics and photomontages lack in both qualities.
Computer Graphics
photomontage
Definition: BicoherenceBicoherence = normalized bispectrum (3rd order moment spectra)Definition (Bicoherence) The bicoherence of a signal x(t)with its Fourier transform being X(ω) is given by:
1 2
*( ( , )1 2 1 2
1 2 1 22 21 2 1 2
[ ( ) ( ) ( )]( , ) ( , )
[ ( ) ( ) ] [ ( ) ]j bX
X X
E X X Xb b e
E X X E Xω ωω ω ω ω
ω ω ω ωω ω ω ω
Φ+= =
+
MagnitudePhase
Numerator =Bispectrum
Normalized by the Cauchy-Schwartz Inequality upper bound
2
2 2
Cauchy-Schwartz Inequality
Hilbert space, : is a random variable satisfying [ ]
[ ] [ ] [ ] ( , )
x x E x
E xy E x E y x y∗
Κ= <∞
≤ ∈Κ
Notations:
( )magnitude
phase⋅ =
Φ ⋅ =
Why BIC is Good for Splicing Detection? Hypothesis I [Farid99]
Quadratic Phase Coupling (QPC)A phenomena where quadratic related frequencies
has the same quadratic relationship1 2 1 2, and ω ω ω ω+1 2 1 2, and φ φ φ φ+ Phases are quadratic coupled
(not independent)!
*1 2 1 2
1 2 2 21 2 1 2
[ ( ) ( ) ( )]( , )[ ( ) ( ) ] [ ( ) ]
X
X X
E X X XbE X X E X
ω ω ω ωω ωω ω ω ω
+=+
If (ω1 , ω2 , ω1+ω2) are quadratic phase coupled,
1 2*
1 2 1 2
1 2 1 2
1 2 1 2
1 2
1) [ ( , )] would be 0
[ ( ) ( ) ( )][ ( )] [ ( )] [ ( )]
( ) 02) ( , ) would be close to unity (imagine X now becomes positive RV)
b
X X XX X X
b
ω ω
ω ω ω ωω ω ω ω
φ φ φ φω ω
Φ
Φ +=Φ +Φ −Φ += + − + =
If (ω1 , ω2 ,ω1+ω2) have statistically independent phase,
1 2
1 2
1) ( , ) would be 0 due to statistical averaging2) [ ( , )] would be random
b
b
ω ω
ω ωΦ
Hypothesis I (cont.)
1 1 2 2( ) cos( ) cos( )x t t tω φ ω φ= + + +1 1
1 1 2 22 2
1 2 1 2
1 2 1 2
1 1 2 2
( ) cos(2 2 ) cos(2 2 )cos(( ) ( ))cos(( ) ( ))
cos( ) cos( ) 1
y t t ttt
t t
ω φ ω φω ω φ φω ω φ φ
ω φ ω φ
= + + +
+ + + ++ − + − +
+ + + +
Argument [Farid99]: Quadratic-linear operation gives rise to QPC and a nonlinear function, in Taylor expansion, contains quadratic-linear term. As splicing is a nonlinear operation, hence bicoherence is good at detecting splicing.
Problems:1. No detailed analysis was given. 2. The quadratic-linear operation here is a point-wise operation, it is not
clear how splicing can be related to a point-wise operation?
2
Quadratic linear Operation( ) ( ) ( )y t x t x t= +
OutlineReview
Problem and MotivationOur ApproachDefinition: BicoherenceWhy Bicoherence good for splicing? Quadratic Phase Coupling Hypothesis
Bicoherence FeaturesMagnitude featurePhase feature
Proposed Image Splicing ModelBipolar Perturbation HypothesisBicoherence of bipolar signalBipolar perturbation effect on magnitude featureBipolar perturbation effect on phase feature
Columbia Image Splicing Detection Evaluation Dataset
933 authentic and 912 spliced image blocks (128x128 pixels)Extracted from
Berkeley’s CalPhotos images (contributed by photographers) which we assume to be authentic
Splicing is done by cut-and-paste of arbitrary-shaped objects and also vertical/horizontal strip.
Authentic
Samples
Spliced
TexturedSmooth
TexturedTextured
SmoothSmoothTextured Smooth
Download URL: http://www.ee.columbia.edu/dvmm/newDownloads.htm
Definition: Phase HistogramPhase histogram (normalized)
1 22
1 21 2 1 2 1 2
1( ) 1 [ ( , )] , ,...,
where 1 1 otherwise 0
2 2 , | , ; , 0,..., 1
(2 1) (2 1) | (2 1) (2 1)
i i
i
p b i N NM
truem m m m M
M Mi iN N
ω ω
π πω ω ω ω
π πφ φ
ΩΨ = Φ ∈Ψ =−
=
Ω= = = = −
− +Ψ = ≤ ≤+ +
∑
Strong phase concentration at ±90°
Symmetry Property: For real-valued signal, bicoherence phase histogram is symmetrical, i.e.,
( ) ( )i ip p −Ψ = Ψ
Bicoherence FeaturesDefinition: Phase feature
Definition: Magnitude feature
( ) log ( ) where ( ) is phase histogramP i i iif p p p= Ψ Ψ Ψ∑
1 21 22 ( , )
1 ( , )Mf bM ω ω
ω ω∈Ω
= ∑
Additional Results on BicoherenceFeatures [ISCAS’04]
0.50.55
0.60.65
0.70.75
0.80.85
0.9
Accuracy Mean Average Percision Average Recall
Plain BICPrediction ResiduePlain BIC + Prediction ResiduePlain BIC + EdgePrediction Residue + EdgePlain BIC + Prediction Residue + Edge
72%62%
OutlineReview
Problem and MotivationOur ApproachDefinition: BicoherenceWhy Bicoherence good for splicing? Quadratic Phase Coupling Hypothesis
Bicoherence FeaturesMagnitude featurePhase feature
Proposed Image Splicing ModelBipolar Perturbation HypothesisBicoherence of bipolar signalBipolar perturbation effect on phase featureBipolar perturbation effect on magnitude feature
Hypothesis II: Bipolar perturbation model
Original image signal is relatively smooth due to the low-pass anti-aliasing operation in camera or scanner.Spliced image signal can have arbitrary discontinuity
difference
Definition (Bipolar signal):
1 2 1 2
1 2
( ) ( ) ( ) ( ) exp( ) exp( ( ) )
where ( ) is a delta function, 0, 0
o o o oFouriertransform
d x k x x k x x D k jx k j x
k k
δ δ ω ω ω
δ
= − + − −∆ ⇔ = − + − +∆
⋅ < ∆>
( )d x
( )s x
( )
( ) ( )ps x
s x d x= +
( ) ( )ps x s x−k1
k2∆
xo
Bicoherence of Bipolar Signal
Results: Bicoherence phase of bipolar signal is concentrated at ±90°:
When there is phase coherency, bicoherence magnitude is close to unity
[ ]31 2 1 2 1 1 1 2( ) ( ) ( ) 2 sin( ) sin( ) sin( ( ))D D D k jω ω ω ω ω ω ω ω∗ + = ∆ + ∆ − ∆ +
Resulting in ±90° phase bias
Bipolar Perturbation Effect on Phase Feature
Bipolar perturbation( ) ( ) ( ) ( ) ( ) ( )p pFourier
Transform
s x s x d x S S Dω ω ω= + ⇔ = +
[ ]1 2 1 2 1 2 1 2
31 2 1 1 1 2
( ) ( ) ( ) ( ) ( ) ( )
( , , , ) 2 sin( ) sin( ) sin( ( ))p p PS S S S S S
C k k j
ω ω ω ω ω ω ω ω
ω ω ω ω ω ω
∗ ∗+ = + +
∆ + ∆ + ∆ − ∆ +
Numerator of the perturbed signal bicoherence:
Cross term involves both S(ω) and D(ω), hence we
assume that it has no consistent phase across all (ω1, ω2) frequency pair
Consistently contributing to the ±90° phase, for every (ω1, ω2) frequency pair.
The contribution depends on k, the magnitude of the bipolar
Empirical Support for Bipolar Perturbation Model
Spliced averaged phase histogram - Authentic averaged phase histogram
More Spliced image blocks have large phase feature value
Spliced average phase histogram hasSignificantly greater 90 deg phase bias
Effect of Bipolar Perturbation on Magnitude Feature
])()(
[])]()(
[)]()(
[[
)]]()(
[)]()(
[)]()(
[[
),(2
21*212
2
22
114
21*21
*
22
113
21
ωωωω
ωω
ωω
ωωωω
ωω
ωω
ωω
+++
+⋅+
+++
⋅+⋅+
=
Gk
SkEG
kS
Gk
SkE
Gk
SG
kS
Gk
SkE
b
( )( ) ( ) ( ) ( ) ( ) ( ) ( ( ))p pFourierTransform
Ss x s x d x S S D k Gkωω ω ω ω= + ⇔ = + = +
Markov Inequality:[ ( ) ]( )p( )
E SSk k
ωω εε
≥ ≤ For energy signal (finite power)
( )lim ( ) 0k
Spkω ε
→∞≥ =
2( )S ω <∞∑
Effect of Bipolar Perturbation on Magnitude Feature (cont.)
0]|)([|]|)]()([|
|)]()()([|),(lim
221
*221
21*
2121 =
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛≥
+
+−
∞→ε
ωωωω
ωωωωωω
DEDDE
DDDEbP
k
( )lim ( ) 0k
Spkω ε
→∞≥ =
Close to 1, due to phase coherency of bipolar signal
More Spliced image blocks have large magnitude feature value
ConclusionsWe propose a bipolar perturbation model for explaining the effectiveness of bicoherence in detecting image splicingThe prediction of the model matches empirical observations (90 deg phase bias)Columbia Dataset for Image Splicing Detectionhttp://www.ee.columbia.edu/dvmm/newDownloads.htmRecent related work in using image phase information for estimating perceptual image blur:
Local phase coherence and the perception of blurZ Wang and E P Simoncelli.Neural Information Processing Systems, December 2003 (NIPS 2003).
Proof of Markov Inequality
[ ]1( ) ( ) ( ) ( )x x
x E xp x p x dx p x dx x p x dx
ε ε
εε ε ε
+∞
≥ ≥ −∞
≥ = ≤ ≤ =∫ ∫ ∫
Proof of Cauchy-Schwartz Inequality
2 2 22
2 2 2
2 , 0, where is a scalarNote, the above expression is a quadratic polynomial of Then:
4 , 4 0
, , ,
tf g t f t f g g tt
f g f g
f g f g f f g g
+ = + + ≥
− ≤
≤ =
Effect of Bipolar Perturbation on Magnitude Feature
Recall the correlation of bipolar signal:
[ ]31 2 1 2 1 1 1 2( ) ( ) ( ) 2 sin( ) sin( ) sin( ( ))D D D k jω ω ω ω ω ω ω ω∗ + = ∆ + ∆ − ∆ +
( ) exp( ) exp( ( ) ) exp( )(1 exp( ))o o oD k jx k j x k jx jω ω ω ω= − + − +∆ = − − − ∆
In ideal case, if bipolar signal at every segment in the averaging term is identical (having same k, xo and ∆) …..
Then, for every (ω1, ω2) frequency pair:2 2*
1 2 1 2 1 2 1 2
1 2 1 2
1 2
[ ( ) ( ) ( )] [ ( ) ( ) ] [ ( ) ]
( ) ( ) ( ) where c=constant( , ) 1
E D D D E D D E D
D D cDb
ω ω ω ω ω ω ω ω
ω ω ω ωω ω
+ = +
⇔ = +⇒ = Reach Cauchy-Schwartz
Inequality Upper BoundBicoherence
magnitude is 1!
Goal: Image Splicing Detection using Natural-imaging Quality (NIQ)
NIQ: Authentic images comes directly from camera and have low-pass property due to camera optical anti-aliasing low-passDeviations from NIQ: Image splicing introduces arbitrarily rough edges/discontinuities in image signalWe characterize such NIQ using bicoherence
Extraction of BIC Features from image
128128-points DFT(with zero padding and Hanning windowing)
⎟⎠⎞
⎜⎝⎛ +⎟⎠⎞
⎜⎝⎛
+=
∑∑
∑
k kk kk
k kkk
Xk
XXk
XXXkb
221
221
21*
21
21
)(1)()(1
)()()(1
),(ˆ
ωωωω
ωωωωωω
2 21 1( ) ( )h v
Horizontal VerticalMi MiN Ni i
M f f= +∑ ∑
2 21 1( ) ( )h v
Horizontal VerticalPi PiN Ni i
P f f= +∑ ∑
3
4
55
6
6
64Overlapping segments
21
Negative Phase Entropy (P)
( ) log ( )P n nnf p p= Ψ Ψ∑
21 2
11 2( , )
Magnitude mean, ( , )Mf bω ω
ω ω∈ΩΩ
= ∑
*
* To reduce noise effect, phase histogram is obtained from the BIC components with magnitude exceeding a threshold