Gary MarsdenSlide 1University of Cape Town Interfacing Processor and Peripherals Overview.
Being&JohnMalkovichgrail.cs.washington.edu/malkovich/eccv10poster.pdfBeing&JohnMalkovich...
1
Being John Malkovich Ira KemelmacherShlizerman 1 Aditya Sankar 1 Eli Shechtman 2 Steven M. Seitz 1,3 1 University of Washington 2 Adobe Systems 3 Google Inc. “Ever wanted to be someone else? Now you can.” —tagline from the film Being John Malkovich Acknowledgments: This work was supported in part by Adobe and the University of Washington Animation Research Labs. We gratefully acknowledge Jason Saragih for providing the face tracking software. Also, in our experiments we used: videos of Cameron Diaz, George Clooney and John Malkovich downloaded fromYouTube and mefeedia.com a collection of photos of George W. Bush from the LFW face database. Image alignment to canonical pose: The method: Results: Appearance representation: • LBP (Local Binary Pattern) histograms (Ahonen et al 06) • Applied on warped images • Only for mouth & eyes regions • Mouth region divided to 3x5 blocks • Eye region divided to 3x2 blocks Distance measure: Puppeteering evaluation (full measure): Without mouth similarity: Without eyes similarity: Cameron Diaz drives John Malkovich: User drives George W. Bush: (870 photos in Bush’s dataset) D(i , j ) = d appear (i, j ) + α p d pose (i , j ) + α t d appear (i − 1, j ) d appear (i, j) = α m d m (i, j) + α e d e (i, j) d {m,e} LBP histogram χ 2 distances restricted to the mouth and eyes regions α {m.e} corresponding weights Temporal continuity Pose Appearance d pose (i, j) = L(| Y i − Y j |) + L(| P i − P j |) + L(| R i − R j |) Y − yaw, P − pitch, R − roll L(d) − robust logistic normalization function Appearance: Pose: Temporal continuity: appearance dist. between frame i 1 and j The distance between input frame i and target frame j is : 2D aligned: Warped to frontal pose: Photo collections: Face and fiducial points detection (Everingham et al 06) Webcam/Video seq.: Realtime tracking (Saragih et al 09) Align and warp using 3D neutral face • A fully automatic realtime framework that combines a number of face processing components in a novel way • Works with any unstructured photo collection and/or video sequence • No training, or labeling Use your face to drive someone else. Our contribution: Output: Photo of person B with similar facial expression and pose Input: Photo of person A Input: Webcam Output:
Transcript of Being&JohnMalkovichgrail.cs.washington.edu/malkovich/eccv10poster.pdfBeing&JohnMalkovich...
Being John Malkovich Ira Kemelmacher-‐Shlizerman1 Aditya Sankar1 Eli Shechtman2 Steven M. Seitz1,3
1 University of Washington 2 Adobe Systems 3 Google Inc.
“Ever wanted to be someone else? Now you can.” —tagline from the film Being John Malkovich
Acknowledgments: This work was supported in part by Adobe and the University of Washington Animation Research Labs. We gratefully acknowledge Jason Saragih for providing the face tracking software. Also, in our experiments we used: -‐ videos of Cameron Diaz, George Clooney and John Malkovich downloaded from YouTube and mefeedia.com -‐ a collection of photos of George W. Bush from the LFW face database.
Image alignment to canonical pose:
The method:
Results:
Appearance representation: • LBP (Local Binary Pattern) histograms (Ahonen et al 06) • Applied on warped images • Only for mouth & eyes regions • Mouth region divided to 3x5 blocks • Eye region divided to 3x2 blocks
Distance measure: Puppeteering evaluation (full measure):
Without mouth similarity:
Without eyes similarity:
Cameron Diaz drives John Malkovich:
User drives George W. Bush: (870 photos in Bush’s dataset)
€
D(i, j) = dappear (i, j) +α pdpose (i, j) +α tdappear (i −1, j)
€
dappear (i, j) = αmdm (i, j) +α ede (i, j)
d{m,e} -‐ LBP histogram χ2 distances
restricted to the mouth and eyes regions
α{m.e} -‐ corresponding weights
Temporal continuity Pose Appearance
€
dpose (i, j) = L(|Yi −Yj |) + L(|Pi − Pj |) + L(|Ri − R j |)Y − yaw, P − pitch, R − roll
L(d) − robust logistic normalization function
Appearance: Pose:
Temporal continuity:
€
appearance dist. between frame i -‐1 and j
€
The distance between input frame i and target frame j is :
2D aligned:
Warped to frontal pose:
Photo collections: Face and fiducial points detection (Everingham et al 06)
Webcam/Video seq.: Real-‐time tracking (Saragih et al 09)
Align and warp using 3D neutral face
• A fully automatic real-‐time framework that combines a number of face processing components in a novel way
• Works with any unstructured photo collection and/or video sequence
• No training, or labeling
Use your face to drive someone else. Our contribution:
Output: Photo of person B with similar facial expression and pose
Input: Photo of person A
Input: Webcam Output:
![arXiv:1909.07957v1 [cs.CV] 17 Sep 2019 · 2019. 9. 18. · Haotian Zhang1 Long Mai 2Ning Xu Zhaowen Wang 2John Collomosse;3 Hailin Jin2 1Stanford University 2Adobe Research 3University](https://static.fdocuments.us/doc/165x107/605448112273e60f6b149d78/arxiv190907957v1-cscv-17-sep-2019-2019-9-18-haotian-zhang1-long-mai-2ning.jpg)
