Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle ... · Clustering-Based, Fully Automated...
Transcript of Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle ... · Clustering-Based, Fully Automated...
Clustering-Based, Fully Automated Mixed-BagJigsaw Puzzle Solving
Zayd Hammoudeh Chris Pollett
Department of Computer ScienceSan José State University
San José, CA USA
17th International Conference on Computer Analysis of Images and PatternsAugust 22-24, 2017
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
1Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
IntroductionJigsaw Puzzles
I First jigsaw puzzle introduced in the 1760s
I First computational jigsaw puzzle solver introducedin 1964 [4]
I Solving a jigsaw puzzle is NP-complete [1, 3].
I Example Applications: DNA fragment reassembly,shredded document reconstruction, and speechdescrambling
I Generally, the ground-truth source is unknown.
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
2Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
IntroductionMixed-Bag Puzzles
Jig Swap Puzzles: Variant of the traditional jigsaw puzzle
I All pieces are equal-sized squares.
I Piece rotation, puzzle dimensions, and ground-truth inputcontents are all unknown.
“Mixed-Bag”: Simultaneous solving of multiple jig swap puzzles
I The number of inputs may be unknown.
Randomized Solver Input – 2,017 Pieces
Solver Output #1 Solver Output #2 Solver Output #3805 Pieces 540 Pieces 672 Pieces
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
3Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Summary of Key Contributions
I Primary Contribution: Novel mixed-bag puzzle solverthat outperforms the current state of the art [6] by:
I Requiring no external “oracle” information
I Generating superior reconstructed outputs
I Supporting more simultaneous inputs
I Additional Contribution: Define the first metrics thatquantify the quality of outputs from a multi-puzzle solver
Our Contribution:
The Mixed-Bag Solver
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
4Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Mixed-Bag SolverOverview
Basis of the Mixed-Bag Solver: Human puzzle solvingstrategy to:
I Correctly assemble small puzzle regions (i.e., segments)I Iteratively merge smaller regions to form larger ones
Simplified Algorithm Flow:
...
Mixed Bag
Final Assembly
Hierarchical Segment
ClusteringSegmentation Stitching
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag Solver5Segmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
SegmentationMixed-Bag Solver Stage #1
I Segment: Partial puzzle assembly where this is a highdegree of confidence pieces are placed correctly
I Each piece is assigned to at most one segment.
I Role of Segmentation: Provide structure to the set ofpuzzle pieces by partitioning them into disjoint segments
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag Solver6Segmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
SegmentationAlgorithm Overview
I Iterative process consisting of one or more rounds.
I In each round, any pieces not already assigned to asegment pieces are assembled into a single puzzle.
I This assembly is then segmented based on inter-piecesimilarity (i.e., the “best buddies” principle).
I Segments of sufficient size are saved for use in laterMixed-Bag Solver stages.
I Segmentation terminates when an assembly has nosegments whose size exceeds a minimum threshold(e.g., 7).
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag Solver7Segmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
SegmentationFirst-Round Example
Ground-TruthInputs Solver Output Segmented Output
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
8Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
StitchingMixed-Bag Solver Stage #2
I Role of Stitching: Quantify the extent that any pair ofsegments is related
I Mini-Assembly: Places a pre-defined, fixed number(e.g., 100) of pieces
I Stitching Piece: A piece near the boundary of a segmentthat is used as the seed of a single mini-assembly
I Segment Overlap: Inter-segment affinity score based onthe composition of a segment’s mini-assembly
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
9Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
StitchingExample – Single Input Image
Ground Truth Segmenter OutputStitching
PiecesMini-Assembly
Stitching piece selected from upper-right corner of the top segment
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
10Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Hierarchical Segment ClusteringMixed-Bag Solver Stage #3
I A single ground-truth image may be comprised of multiplesegments.
I Role of Hierarchical Clustering: Estimate the number ofinputs by grouping together all segments from the sameground-truth image.
I Single-Link Clustering: Inter-cluster similarity equals thesimilarity of their most similar respective members
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
11Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Terminating the SolverBuilding the final outputs
I The solver continues merging segment clusters until oneof two criteria is satisfied:
I Only a single segment cluster remains
I Maximum similarity between any segment clusters is belowa predefined threshold
I Final Assembly: Builds the final solver outputs are builtusing the cluster membership results
Quantifying Solver Performance
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
12Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Quantifying Solver Performance
I Metrics quantify the quality of the solver outputs as thereconstructions may not be reconstructions.
I Two Primary Quality Metrics: Range [0,1]I Direct Accuracy
I Neighbor Accuracy (not discussed in this presentation)
I Disadvantages of Current Metrics: Neither account forissues unique to mixed-bag puzzles including:
I Pieces from one input misplaced in multiple output puzzles
I Pieces from multiple inputs in the same output
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying Quality13Direct Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Direct AccuracyOverview of the Current Standard
Standard Direct Accuracy: Fraction of pieces, c placed in thesame location in both the ground-truth and solved puzzlesversus the total number of pieces, n
Formal Definition:
DA =cn
(1)
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying Quality14Direct Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Direct AccuracyShiftable Enhanced Direct Accuracy Score (SEDAS)
SEDAS: A new quality metric with two primary improvementsover standard direct accuracy:
SEDASPi = maxl∈L
(maxSj∈S
ci,j,l
ni +∑
k 6=i(mk,j)
)(2)
I Mixed-Bag Support: For input, Pi ∈ P, and output, Sj ∈ S,penalize for missing pieces (via ni ) and additional pieces(via
∑k 6=i mk,j )
I Shiftable Reference: Shift the direct accuracy referencecoordinate, l within a set of possible puzzle piece locations, L,(l ∈ L), in order to maximize the overall score
Experimental Results
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
15Experimental ResultsInput Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Overview of the Experiments
I Standard Jig Swap Puzzle Experiment Conditions:Defined by Cho et al. (CVPR 2010) [2] and followedby [7, 6, 9, 5]
I Procedure: Randomly select, without replacement, aspecified number of images (between 2 and 5) from the805 piece, 20 image data set [8]
I Two Primary Experiments:
I Estimation of the Ground-Truth Input CountI Comparison of Overall Reconstruction Quality
I Baseline: Current State of the Art - Paikin & Tal(CVPR ’15) [6]
I Our Competitive Disdvantage: Paikin & Tal’s algorithm hadto be provided the number of input puzzles.
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental Results16Input Puzzle Count
Solver Comparison
References
Dept. of Computer ScienceSan José State University
Estimating the Ground-Truth Input CountMultiple Input Puzzles
0 1 2 30
20
40
60
80 75
16
72
4448
4 4
50 50
0 0
60
20 20
0
Size of Input Puzzle Count Error
Freq
uenc
y(%
)
Mixed-Bag Solver’s Input Puzzle Count Error Frequency
2 Puzzles 3 Puzzles 4 Puzzles 5 Puzzles
Puzzle Count Error: Difference between the actual numberof input puzzles and the Mixed-Bag Solver’s estimate
Overall Accuracy: 65%I Iterations with Error Greater than One: 8%I Performance consistent across input count
Over-Rejection of Cluster Mergers: Neverunderestimated the number of input puzzles
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
17Solver Comparison
References
Dept. of Computer ScienceSan José State University
Comparison of Reconstruction QualityPerformance on Multiple Inputs
I Goal: Compare the quality of the outputs from theMixed-Bag Solver (MBS) and Paikin & Tal’s algorithm
I Note: Our Mixed-Bag Solver’s performance when itcorrectly estimated the puzzle count is also shown.
I This is an approximate representation of the performancehad there been optimal hierarchical clustering.
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
18Solver Comparison
References
Dept. of Computer ScienceSan José State University
Comparison of Reconstruction QualityShiftable Enhanced Direct Accuracy Score (SEDAS)
2 3 4 50
0.2
0.4
0.6
0.8
1
Number of Input Puzzles
SE
DA
S
Effect of the Number of Input Puzzles on SEDAS
MBS Correct Puzzle Count MBS All Paikin & Tal
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
19Solver Comparison
References
Dept. of Computer ScienceSan José State University
Performance on Multiple Input PuzzlesResults Summary
I Summary: Our Mixed-Bag Solver significantlyoutperforms the state of the art, Paikin & Tal.
I This is despite their algorithm having a competitiveadvantage by being supplied the number of input puzzles.
I Puzzle Input Count: Our approach shows no significantperformance decrease with additional input puzzles.
I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimating the input puzzle count
I Many of the extra puzzles were relatively insignificant insize.
20
Clustering-Based,Fully Automated
Mixed-Bag JigsawPuzzle Solving
Hammoudeh & Pollett
Introduction
Mixed-Bag SolverSegmentation
Stitching
Hierarchical Clustering
Quantifying QualityDirect Accuracy
Experimental ResultsInput Puzzle Count
Solver Comparison
20References
Dept. of Computer ScienceSan José State University
List of References I
[1] Tom Altman. Solving the jigsaw puzzle problem in linear time. Applied Artificial Intelligence, 3(4):453–462, January 1990.ISSN 0883-9514.
[2] Taeg Sang Cho, Shai Avidan, and William T. Freeman. A probabilistic image jigsaw puzzle solver. In Proceedings of the2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’10, pages 183–190. IEEEComputer Society, 2010.
[3] Erik D. Demaine and Martin L. Demaine. Jigsaw puzzles, edge matching, and polyomino packing: Connections andcomplexity. Graphs and Combinatorics, 23 (Supplement):195–208, June 2007.
[4] H. Freeman and L. Garder. Apictorial jigsaw puzzles: The computer solution of a problem in pattern recognition. IEEETransactions on Electronic Computers, 13:118–127, 1964.
[5] Andrew C. Gallagher. Jigsaw puzzles with pieces of unknown orientation. In Proceedings of the 2012 IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), CVPR ’12, pages 382–389. IEEE Computer Society, 2012.
[6] Genady Paikin and Ayellet Tal. Solving multiple square jigsaw puzzles with missing pieces. In Proceedings of the 2015IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’15. IEEE Computer Society, 2015.
[7] Dolev Pomeranz, Michal Shemesh, and Ohad Ben-Shahar. A fully automated greedy square jigsaw puzzle solver. InProceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’11, pages 9–16.IEEE Computer Society, 2011.
[8] Dolev Pomeranz, Michal Shemesh, and Ohad Ben-Shahar. Computational jigsaw puzzle solving.https://www.cs.bgu.ac.il/~icvl/icvl_projects/automatic-jigsaw-puzzle-solving/, 2011. (Accessed on05/01/2016).
[9] Dror Sholomon, Omid David, and Nathan S. Netanyahu. A genetic algorithm-based solver for very large jigsaw puzzles.In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’13, pages1767–1774. IEEE Computer Society, 2013.