Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai...
-
Upload
sarahi-hurford -
Category
Documents
-
view
215 -
download
0
Transcript of Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai...
![Page 1: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/1.jpg)
Improved Seam Carving for Video Retargeting
Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume 27 Issue 3, August 2008Presenter: Hon-Hang ChangDate: 2011/ 09/ 30
1
![Page 2: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/2.jpg)
Outline• Introduction• Preliminaries• Seam Carving using Graph Cut• Forward Energy• Results• Conclusion
2
![Page 3: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/3.jpg)
To treat video as 3D cube
Introduction 1 D path on 2D image (Seam)
2D manifolds in a 3D volume
• In this paper the author define a new formulation of seam carving using graph cuts.
• Because of human perception, the temporal may even be more disturbing than spatial in video, as the human eye is highly sensitive to movement.
• 2D manifolds
3
![Page 4: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/4.jpg)
IntroductionA seam is a monotonic and connected path of pixels going from the top of the image to the bottom
Or from left to right
• The image size is reduced by one either in the horizontal or the vertical dimension.
• video should support retargeting capabilities as it is displayed on TVs, computers, cellular phones.
• A whole seam
4
![Page 5: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/5.jpg)
Preliminaries
Among this, {It} t=1~N
Extend the spatial L1-norm to a spatiotemporal L1-norm:
• Spatiotemporal L1-norm
Spatial term
Temporal term
Where,α ∈ [0, 1]
• Motion artifacts are more noticeable• Taking α= 0.3
5
|}),(||),({|max),(1
jiIy
jiIx
jiE tt
N
tspatial
|}),({|max),(1
jiIt
jiE t
N
ttemporal
temporalspatialglobal EEjiE )1(),(
![Page 6: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/6.jpg)
Seam Carving using Graph Cuts
S (Source)
T (Sink)
Source Sink
arc1
arc2
arc3
arc4
• Graph cut
6
![Page 7: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/7.jpg)
Seam Carving using Graph Cuts
An optimal seam must be:1.1.Valid
- A simple cut cannot define a seam carving2. Monotonic - Including only one pixel in each row (or column)3. Connected -The seam must be connected
• The proposed formulation extend seam carving to video and define a monotonic and connected 2D manifold seam inside the video cube.
• Optimal seam in graph cut
7
![Page 8: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/8.jpg)
Seam Carving using Graph Cuts• Graph cut for image (Non-monotonic)
• Every internal node P is connected to its four neighbors Nbr(Pi,j) = {Pi−1,j , Pi+1,j , Pi,j−1, Pi,j+1}.
Pi,j
Pi-1,j
Pi,j-1 Pi,j+1
Pi+1,j
Neighbors pixels
∂x(i, j) = |I(i, j + 1) − I(i, j)|
∂y(i, j) = |I(i+1, j ) − I(i, j)|
Horizontal direction
Vertical direction
8
![Page 9: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/9.jpg)
Seam Carving using Graph Cuts• Graph cut for image
(Unconnected)
• Prove_1:
9
![Page 10: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/10.jpg)
Seam Carving using Graph Cuts
E1(i, j) =∂x(i, j) + ∂y(i, j)
• Graph cut for image (Original)
• Prove_2:
10
![Page 11: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/11.jpg)
Seam Carving using Graph Cuts• Graph cut for image
• We can use any energy function defined on the pixels as the weight of the forward horizontal arcs .
• Achieve the same results as the original dynamic programming based seam carving
11
![Page 12: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/12.jpg)
Seam Carving using Graph Cuts• Graph cut for video
• The extension to video is straightforward. 12
![Page 13: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/13.jpg)
Seam Carving using Graph Cuts• Graph cut for video
• Computation time is quadratic in the number of voxels. O(mn2)- n #of nodes ; m#of arcs
• Performance issues are encountered already for high resolution images
• The graph cut approach to seam carving allows us to extend the benefits of content-aware resizing to video. Still, the method is not perfect and no single energy function was shown to perform properly in all cases. Therefore, they introduce a new energy function that better protects media content, and improves video results.
13
![Page 14: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/14.jpg)
Forward Energy
• Remove the seam with the least amount of energy from the image
• The inserted energy is due to new edges created by previously non adjacent pixels that become neighbors once the seam is removed
• To measure the real change in energy after a removal of a seam
They measure the difference in the energy of the image after the removal (It=i+1) and the energy of only those parts that were not removed in the previous image
14
![Page 15: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/15.jpg)
Forward Energy
15
![Page 16: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/16.jpg)
+LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1)difference between the Left and Right neighbors
+LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j)difference between the Left and Up neighbors
-LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j)difference between the Left and Up neighbors with respect to the end point of the arrow
Forward Energy• Forward Energy in Graph Cut
16
![Page 17: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/17.jpg)
Forward Energy• Forward Energy in Graph Cut
+LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1)difference between the Left and Right neighbors
+LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j)difference between the Left and Up neighbors
-LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j)difference between the Left and Up neighbors with respect to the end point of the arrow 17
![Page 18: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/18.jpg)
Results
18
![Page 19: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/19.jpg)
Results
19
![Page 20: Improved Seam Carving for Video Retargeting Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume.](https://reader030.fdocuments.us/reader030/viewer/2022032701/56649c8e5503460f949474d2/html5/thumbnails/20.jpg)
Conclusion
20
Video retargeting is achieved using graph cuts and we have shown a construction that is consistent with the dynamic programming approach.