To appear in ACM Transactions on Graphics

Exposing Photo Manipulation with Inconsistent ShadowsEric Kee, Dartmouth CollegeJames OBrien, University of California, Berkeleyand

Hany Farid, Dartmouth College

We describe a geometric technique to detect physically inconsistent ar-rangements of shadows in an image. This technique combines multiple con-straints from cast and attached shadows to constrain the projected locationof a point light source. The consistency of the shadows is posed as a linearprogramming problem. A feasible solution indicates that the collection ofshadows is physically plausible, while a failure to find a solution providesevidence of photo tampering.

Categories and Subject Descriptors: I.2.10 [Artificial Intelligence]: Visionand Scene UnderstandingScene Analysis; I.3.6 [Computer Graphics]:Methodology and TechniquesImage Forensics; K.4.m [Computing Mi-lieux]: Computers and SocietyMiscellaneous

Additional Key Words and Phrases: Shadows, image forensics, photo ma-nipulation, image manipulation, forgery detection

ACM Reference Format:Kee, E., OBrien, J. F., and Farid, H., 2013. Exposing Photo Manipulationwith Inconsistent Shadows. ACM Trans. Graph. ??, ?, Article ??? (August2013), 12 pages.To appear.

1. INTRODUCTION

Recent advances in computational photography, computer vision,and computer graphics allow for the creation of visually com-pelling photographic fakes. The resulting undermining of trust inphotographs impacts law enforcement, national security, the me-dia, advertising, e-commerce, and more. The nascent field of photoforensics has emerged to help restore some trust in digital pho-tographs [Farid 2009].

In the absence of any digital watermarks or signatures, tech-niques in image forensics operate on the assumption that mostforms of tampering will disturb some statistical or geometric prop-erty of an image. In a well-executed forgery these disturbances

Authors email addresses: erickee@cs.dartmouth.edu, job@berkeley.edu,farid@cs.dartmouth.eduPermission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesshow this notice on the first page or initial screen of a display along withthe full citation. Copyrights for components of this work owned by othersthan ACM must be honored. Abstracting with credit is permitted. To copyotherwise, to republish, to post on servers, to redistribute to lists, or to useany component of this work in other works requires prior specific permis-sion and/or a fee. Permissions may be requested from Publications Dept.,ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax+1 (212) 869-0481, or permissions@acm.org.c 2013 ACM 0730-0301/2013/13-ARTXXX $10.00

DOI: http://dx.doi.org/10.1145/XXXXXXX.YYYYYYY

Original image copyright 1969, NASA

Fig. 1. Our algorithm finds that the shadows in this 1969 moon landingphoto are physically consistent with a single light source. The solid linescorrespond to constraints from cast shadows and dashed lines correspondto constraints from attached shadows. The region outlined in black, whichextends beyond the figure boundary, contains the projected light locationsthat satisfy all of these constraints.

will either be perceptibly insignificant or they may be noticeablebut subjectively plausible. Methods for forensic analysis provide ameans to detect and quantify specific types of tampering. To theextent that these perturbations can be quantified and detected, theycan be used to objectively invalidate a photo.

We describe a technique for determining if cast and attachedshadows in a photo are consistent with the model of a single distantor local point light source. Forensic techniques based on analyz-ing lighting and shadows are attractive because 3-D lighting effectscan be difficult to modify using commercial photo editing software,and low quality images can be analyzed since lighting effects andshadows survive common operations such as image compressionand down-sizing.

There is some evidence that the visual system is capable of de-tecting small changes in lighting direction in simple controlled set-tings [Koenderink et al. 2004; Mamassian 2004; Khang et al. 2006;Pont and Koenderink 2007; Koenderink et al. 2007; OShea et al.2010]. In more complex settings, however, the visual system is farless capable at detecting gross inconsistencies in lighting [Jacobsonand Werner 2004; Ostrovsky et al. 2005; Farid and Bravo 2010]. In

ACM Transactions on Graphics, Vol. 32, No. 4, Article XXX, Publication date: September 2013.

XXX:2

a forensic setting, a multitude and variety of cast and attached shad-ows from complex shapes are cast onto equally complex and variedsurfaces. Such limitations of the visual system imply that a forgermay overlook inconsistencies in lighting and shadowing, and a vi-sual inspection of shadows will, at best, be highly subjective.

The iconic photo of the 1969 moon landing, Fig. 1, provides anexample of the variety and complexity of shadows that are com-mon in photos. In fact, it has been argued by conspiracy theoriststhat the shadows in this photo are physically implausible and henceevidence of photo tampering and broader nefarious conspiracies.Beyond a subjective visual analysis, the physical consistency ofshadows can be objectively determined by considering their basicgeometry.

Consider a ray that connects a point in a shadowed region toits corresponding point on the shadow-casting object. In the 3-Dscene, this ray intersects the light source. In a 2-D image of thescene created under linear perspective, the projection of this rayremains a straight line that must connect the images of the shadowpoint and object point, and intersect the projected image of the lightsource. These constraints hold regardless of the geometry of the ob-ject and the surface onto which the shadows are cast, and for eitheran infinitely distant or local light. Multiple constraints can there-fore be used to determine the projected location of a light source inthe image plane. Note that this projected location corresponds to aninfinite number of 3-D light positions. We concern ourselves withthe 2-D projection because a single image typically does not allowone to compute the 3-D location of the light in the scene.

If a scene purportedly contains a single light source but the shad-ows in the scene specify mutually inconsistent constraints that can-not be satisfied by any single light position, then this inconsistencyevidences photo tampering. It can sometimes be difficult or impos-sible to precisely match a point on a shadow to its correspondingpoint on an object particularly on attached shadows that form asa surface smoothly curves to face away from the light. We thereforeconsider a relaxed, conservative constraint in which the location ofpoints on the object are restricted to a range of possible locations.These relaxed constraints specify either angular wedges or half-planes in the image that restrict the projected location of the lightsource. (See Fig. 2.) The consistency of multiple such constraintsis framed as a linear programming problem. A viable solution isinterpreted to mean that the shadows are physically plausible whilea failure to find a solution is used as evidence of photo tampering.

1.1 Related Work

Previous lighting-based forensics methods estimate the 2-D light-ing direction or lighting environment from the shading on an ob-jects contour [Johnson and Farid 2005; 2007]. If the 3-D geometryof an object is known, then the 3-D lighting direction or lightingenvironment can be estimated [Kee and Farid 2010]. Related com-puter vision techniques that estimate lighting from a single imageuse object shading [Nillius and Eklundh 2001], or shadows castonto planar surfaces [Sato et al. 2003; Okabe et al. 2004; Lalondeet al. 2011]. In [Karsch et al. 2011], manually approximated scenegeometry is used to fit a local lighting model that is perceptuallyplausible, but insufficient for forensic application because the phys-ical accuracy is heavily influenced by user input.

Photometric inconsistencies of a cast shadows umbra were usedto detect inconsistent shadows in [Liu et al. 2011]. Inconsisten-cies in the location of a cast shadow were used in [Zhang et al.2009], but they placed several assumptions on the scene geome-try: shadows were cast onto a planar ground plane and the objectscasting shadows were vertical relative to the ground plane. In the

Original image copyright 2011, Geico Insurance

Fig. 2. Shown are: (top) a frame from the Geico commercial Dunk Easier Way to Save depicting a somewhat incredible athletic performance;(middle) examples of cast (1-2) and attached (3) shadow constraints. Theprojected location of the light source must lie in the intersection of theseconstraints. Shown in the bottom is a multitude of constraints (solid linescorrespond to cast shadows and dashed lines correspond to attached shad-ows). The shadows from the people and house are consistent with a lightsource located somewhere in the black-outlined region. The boys shadow,however, is inconsistent with the rest of the scene.

most closely related work [Stork and Johnson 2006], the consis-tency of cast shadows in artworks was determined by identifying

ACM Transactions on Graphics, Vol. 32, No. 4, Article XXX, Publication date: September 2013.

XXX:3

Fig. 3. A cylinder and sphere illuminated from the left with a distant pointlight. The red line and curve correspond to the terminator the contour atwhich the surface normal is oriented 90 degrees from the direction to thelight source. Points to the right of the terminator are in an attached shadow.

points on a cast shadow and their corresponding po