Model and Dictionary guided Face In-painting in the WildReuben Farrugia (University of Malta) and Christine Guillemot (INRIA Rennes Bretagne)

Proposed Face In-painting Algorithm

Facial occlusion is a critical issue in image editing and forensicscenarios – and can be used to improve face recognition ofoccluded and non-frontal faces.

This work presents a method that can be used to in-paint occludedfacial regions with unconstrained pose and orientation.

This approach first warps the facial region onto a reference modelto synthesize a frontal view (a.k.a. Frontalization)

A modified Robust Principal Component Analysis (RPCA)approach is then used to suppress warping errors.

It then uses a novel local patch-based in-painting algorithm whichhallucinates missing pixels using a dictionary of face images whichare pre-aligned to the same reference model.

The hallucinated region is then warped back onto the originalimage to restore occluded pixels

NTIRE 2016in conjunction with ACCV 2016

Face Frontalization

The face image is first warped using affine transformation. Thewarped image is then stacked with a set of undistorted frontallyaligned training face images to form the augmented matrix Ma.This matrix is then decomposed using the following low-rankoptimization problem

Figure: Schematic diagram of the proposed face In-painting in theWild method

minimize 𝑨 ∗ + 𝜇 𝑬 1 subject to 𝑨 + 𝐄.= 𝑀𝒂

The first column of A corresponds to the de-noised frontal faceimage which removes the warping errors

Figure: The first column from each ell snows the cropped imagefrom the LFW dataset. The second column represents thepiecewise affine warped face image and the third column wererestored using our modified RPCA method.

Face In-Painting

The proposed face in-painting method is based on the classicalapproach of Criminisi. However, instead of using a dictionary ofneighboring patches we use collocated patches from a dictionary ofaligned face images.

The weights w* to reconstruct the unknown part of a patch arecomputed by solving the following optimization problem.

𝒘∗ = argmin𝒘∗ 𝝋𝒑𝒌 − 𝑫𝒑

𝒌 𝒔 𝒘∗𝟐

𝟐𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 σ𝒊=𝟏

𝑲 𝒘∗ = 𝟏

The unknown part of each patch is then reconstructed using

𝝋𝒑𝒖 = 𝑫𝒑

𝒖 𝒔 𝒘∗

Figure: Schematic diagram of the local face in-painting offrontalized face images.

Experimental Results

In this experiment we used 84 images from the AR dataset witchwere occluded using the following masks to mimic real-worldscenarios.

Experimental results demonstrate a significant gain in performanceover existing face in-painting algorithms.11

The proposed method was evaluated on real-world images atdifferent orientations, resolutions and quality including CCTVquality images taken during riots in UK.

Figure: Original image (left), the original image with the region toebe concealed marked in green (center-left), the inpainted imageof the frontal region (center-right), and the final inpainted faceusing off the shelf inpainting method..