study Seam Carving For Content Aware Image Resizing
-
Upload
chiamin-hsu -
Category
Technology
-
view
4.417 -
download
6
description
Transcript of study Seam Carving For Content Aware Image Resizing
Seam Carving for Content-Aware Image Resizing
Shai Aidan (Mitsubishi Electric Research Labs)
Ariel Shamir (The Interdisciplinary Center & MERL)
ACM SIGGRAPH 2007
Resize
Seam carving & insertion
AbstractSeams are optimal 8-connected paths of
pixels cross the imageCarving out or inserting seams to achieve
content-aware resizing
OutlineIntroductionBackgroundSeam-carving operatorDiscrete image resizingMulti-size imagesLimitationsConclusions and future work
INTRODUCTION
MotivationHTML can support dynamic
changes of page layout and text. Why can not an image deform to fit different layout automatically ?◦ iGoogle
How about aspect ratio of an image , such as fitting photo into PDA or phone cells ?
Solution ?◦ Resize – content independent◦ Crop – remove pixels from the image
periphery only
Basic Idea of Seam-CarvingUse energy function to define the
importance of pixelsDefine seam-carving image operator
◦ Image reduction Carving out seams - the connected low energy
pixels crossing the image
Preserving the image structure
◦ Image enlarging Insert seams on low energy area The order of seam insertion ensures a balance
between the original image content and the artificially inserted pixels
ApplicationDiscrete image resizing
◦ Aspect Ration Change, Image Retarget, Image Enlarging, Content Amplification, Seam Carving in gradient domain, Object Removal
Multi-size images◦ An image can continuously change their
size in a content-aware manner◦ Storing the order of seam removal and
insertion
BACKGROUND
Image RetargetSeek to change the size of the image
while maintaining the important features ◦ Face detector
An automatic thumbnail creation [Suh 03] ◦ ROI
Fisheye-View warp [Liu and Gleicher 05, 06]◦ Visual saliency []
[Suh 03]
origin
[Selur 04, decompose image to foreground obj and background
Saliency map[Itti IEEE99]
◦Simulate neuroscience of human visual system
◦Pyramid tech. to compute 3 feature maps, color, intensity and orientation
[Suh 03], an automatic thumbnail creation, based on either a saliency map or the output of a face detector
[Chen 03], adapting most important region of images to mobile devices.
[Liu 03], suggesting to trade time for space. Given a collection of regions of interest, they construct an optimal path through these regions and display them serially.
[Santella et al. 06] use eye tracking, in addition to composition rules to crop images intelligently.
ROI (Region-Of-Interest) Such a method was proposed by [Liu
and Gleicher 05, 06] for image and video retargeting. For image retargeting they find ROI and construct a novel Fisheye-View warp that essentially applies a piecewise linear scaling function in each dimension to the image. This way the ROI is maintained while the rest of the image is warped. The retargeting can be done in interactive rates, once the ROI is found, so the user can control the desired size of the image by moving a slider. In their video retargeting work they use a combination of image and saliency maps to find the ROI. Then they use a combination of cropping, virtual pan and shot cuts to retarget the video frames.
Feature-aware warping The first solution to the
general problem of warping an image into an arbitrary shape while preserving user-specified features was recently proposed by [Gal et al. 06].
The feature-aware warping is achieved by a particular formulation of the Laplacian editing technique, suited to accommodate similarity constraints on parts of the domain.
Since local constraints are propagated by the global optimization process, not all the constraints can always be satisfied at once
Seam Perfect seams to combine parts of a set of photo into a single
composite picture [Agarwala et al. 04]
Drag-and-Drop Pasting that extends the Poisson Image Editing to computer an optimal boundary (seam) between the source picture and target images [Jia et al. 06]
AutoCollage, a program that automatically creates a collage image from a collection of images. [Rother et al. 06]
Simultaneously solve matting and compositing. They allow the user to scale the size of the foreground object and paste it back on the original background. [Wang , Cohen 06]
evaluated several cost functions for seamless image stitching and concluded that minimizing an L1 error norm between the gradients of the stitched image and the gradients of the input images performed well in general [Zomet et al. 05]
Sear Optimal SeamDijkstra’s shortest path algorithm
[98]Dynamic programming [Efros 01]Graph cuts [Kwatra 03]
SEAM-CARVING OPERATOR
Stra
teg
ies o
f Imag
e R
ed
uctio
n
Optimal global remove the lowest energy pixels
Pixelremove the least energy in each row
Original
e1 energy
Columnremoving columns with minimal energy
Cropfind a sub-win with the highest energy
Original
e1 energy
Stra
teg
ies o
f Imag
e R
ed
uctio
n
Vertical Seam
Horizontal Seam
Optimal Seam Search
Optimal Seam Search
Dynamic Programming
G
S
e1 energy
Image Energy PreservationThe average energy of all pixels during
resizing
Energy FunctionsL1 and L2-norm of the
gradient, saliency measure [Itti 99]
Histogram of Gradient (HoG)Histogram of Gradient (HoG)
[Dalal and Triggs 95]1. Dividing the image window into cells2. For each cell accumulating a local 1-D
histogram of gradient directions3. Normalize cells by the measure of local
histogram energy over larger blocks
The average gradient image
R-HOG descriptor
Weighted R-HOG descriptor
Energy FunctionsHistogram of Gradient
(HoG) [Dalal and Triggs 95]◦ max(HoG(I(x,y)) makes sure
the seams run parallel to the
edge of objects and not
cross them
Energy FunctionsEntropy
◦ Compute the entropy over a 9 x 9 window and add it to e1
eEntropy(x,y) =
+ e1 (x,y)
Energy FunctionsSegmentation and L1
1. Image segmentation [Christoudias 02]
2. Apply e1 on the results
No single e function performs well across all images
Similar range for resizing
e1 or eHoG works well
DISCRETE IMAGE RESIZING
Aspect Ratio Change, Retargeting with Optimal Seams-Order, Image Enlarging, Content Amplification,
Aspect Ratio ChangeCarving-out /insert seams
Original
Original
Original
1D aspect ratio changing
Optimal Seams-Order Search
Dynamic Programming
+
+
=
min
n x m
n‘ x m’
2D aspect ratio changing
Retargeting with Optimal Seams-Order
optimal
h-first
Original
alternate
Transport map
v-first
Image Enlarging
enlarged image
I(t): smaller image after t seam-carving
I(-k): enlarged image after k seam insertion
I(t)I(-1)
t
I(-k)
I(-k)
1. Find first k seams for removal
2. Duplicate them in order to arrive at
I(-k)
insert seams in order of removal
origin
Image Enlarging (> 50%)1. Break into several
steps2. Each step does not
enlarge the size of image more than a fraction
origin
Content Amplification
Original
Scale
Seam
Carving
Original Amplified
Seam Carving in the Gradient DomainSeam + Poisson
Reconstruction [Perez 03]1. Compute e
function
2. Work on the gradient domain
3. Remove seams from the x and y derivatives of the original image
4. Use Poisson Reconstruction
original retarget
retarget in Gradient Domain
Object Removal
1. Mark the removing target
2. Remove seams until all the marked pixels are gone
3. * Employ seam insertion to maintain the original size
Object RemovalOrigin
Multi-size imagesStore the pre-computed
representation that encodes, for each pixel in V/H map◦ The index of the seam that removed it◦ The negative index of the seam that
inserted it
origin V(i,j)=t : pixel (i,j) removed by t-th vertical seam
H(i,j)=t : pixel (i,j) removed by t-th horizontal seam
Blue (first seam) Red (last seam)
LimitationsSeam-Carving does not work well on all
imagesEx: face
Origin Crop Scale
Bottom up feature detection
Constraint the face
Face the flower
LimitationsThe amount of
content◦ Too density, no
“less” important area
The layout of the image content
origin
origin
ConclusionsPresent a content-aware resizing
using the seam-carving image operator
Seams are the optimal paths on a single image◦ Carve-out seams◦ Insert seams
Application of seam-carving operator◦ Aspect ratio change, image retargeting,
content amplification, object removalMulti-size images that support
continuous resizing in real-time
Future WorkVideo resizingCombination of scaling and seam-
carving◦ Define more robust multi-size image
Better solution to combine horizontal and vertical seams in multi-size image
END