Post on 15-Mar-2020
Computing Nearest-Neighbor Fields via Propagation-Assisted KD-Trees Kaiming He and Jian Sun
Microsoft Research Asia
Introduction Results
• Computing Nearest-Neighbor Fields (NNF) is to densely match patches between two images.
Overview
Source Image A Target Image B
Nearest-Neighbor Field
• The applications of NNF: image inpainting/retargeting [1], texture synthesis, super-resolution, denoising, etc.
• Contributions: a fast and accurate method for computing NNF.
[1] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. Patchmatch: a randomized correspondence algorithm for structural image editing. In SIGGRAPH, 2009 [2] S. Korman and S. Avidan. Coherency sensitive hashing. In ICCV, 2011.
• Our perspective: ANN search for multiple but dependent queries.
Patches in A
Query Set
Patches in B
Candidate Set
• Methodology comparisons:
Method Strategy in Query Set
Strategy in Candidate Set
Traditional LSH (Locality Sensitive Hashing)
None Data-independent
Traditional kd-tree None Data-adaptive
PatchMatch [1] Propagation Random Sampling
CSH [2] (Coherency Sensitive Hashing)
Propagation Data-independent (LSH)
Ours Propagation Data-adaptive (kd-tree)
• Our advantage: better utilizes both the dependency of the patches in A and the dependency of the patches in B.
Algorithm
• Basic Algorithm
1. Build a kd-tree using all the patches in the Candidate Set. Each leaf contains no more than m candidates.
2. Scan the image A in raster order. For each patch pA(x, y) in A, do propagation-assisted kd-tree search as follows:
i. Descend the tree to a leaf (Leaf #0); ii. Propagate a leaf from left/upper, using the already matched
result of pA(x-1, y) and pA(x, y-1); iii. Find the NN of pA(x, y) in all the above leaves.
image Bimage A
Propagation in PatchMatch [1]
(x-1, y)(x, y)
(x’-1, y’)(x’, y’)
image Bimage A
Leaf #1
Propagation-Assisted KD-tree
(x-1, y)(x, y)
(x’-1, y’)(x’, y’)
• Method Comparison: PatchMatch [1]
• Performance in VidPairs Benchmark [2] (Accuracy vs. Time, 133 image pairs)
References
40
45
50
55
60
65
0 5 10 15 20 25 30 35 40 45 50
L 2di
st.
seconds
PMCSHOurs (no re-ranking)Ours (re-ranking)Traditional kd-treeGround Truth
2Mp images, 8x8 patches
Observations: 1. 10-20x faster vs. PatchMatch [1], 2-5x faster vs. CSH [2], at the same accuracy. 2. 70% smaller error vs. PatchMatch, 50% smaller error vs. CSH, at the same running time. 3. Traditional kd-trees can be comparable with PatchMatch.
• Visual Comparisons
Image B
PM CSH GTOurs
Image A
PM’s NNF
CSH’s NNF
Our NNF
Groundtruth
PM’s error
CSH’s error
Our error
NNFs and error maps
CSH GTOursPM
PM CSH GTOurs
PM CSH GTOurs
Reconstructed Images
• Method Comparisons: kd-tree
(a) Standard search(err: 11.13)
e0
e1
1200
120
(b) Priority search(err: 10.71)
e0
e1
1200
120
(d) Propagation from GT(err: 2.77)
e0
e1
1200
120
(c) Randomized kd-trees(err: 7.80)
e0
e1
1200
120
(e) Our propagation search(err: 3.04)
e0
e1
1200
120
e0: error of Leaf #0; e1: error of Leaf #1. Leaf #1 is given by backtracking (standard/priority search), another random tree (randomized kd-tree), or propagation (ours).