sketch to photo matching
-
Upload
chippy-thomas -
Category
Law
-
view
92 -
download
0
Transcript of sketch to photo matching
![Page 1: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/1.jpg)
MET’S School of EngineeringDepartment of Computer Science and Engineering
Academic Year 2015-2016
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 1
SKETCH TO PHOTO MATCHING SYSTEM
Submitted by
Chippy Thomas (GWAOCCS004)
S4-M.Tech CSE (2014-16 Batch)
19-03-2016
![Page 2: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/2.jpg)
May 2, 2023 2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction
System that match sketch with mugshot
For the identification of suspectsUse when eyewitness are only evidence
Sketch/composites-hand-drawn-software generated-surveillance composites
•mugshot19-03-2016 2
![Page 3: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/3.jpg)
May 2, 2023 3
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction – contd… So matching sketch and photo
Direct matching is not possibleIe..We don’t match them pixel by pixel
Match by finding high-level features from both the sketch and the photo
Features such as the structural distribution and the shape of the eyes, nose and chin etc.
19-03-2016 3
![Page 4: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/4.jpg)
May 2, 2023 4
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Introduction – contd…
19-03-2016 3
![Page 5: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/5.jpg)
May 2, 2023 5
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Hand-drawn composite
19-03-2016 4
•Drawn by forensic artist •Based on description provided by one or more witness•Artist need proper training
![Page 6: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/6.jpg)
May 2, 2023 6
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Software-generated composites
19-03-2016 5
•Drawn by operator using software•Most used software's are identikit and FACES•Operator need training
![Page 7: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/7.jpg)
May 2, 2023 7
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Surveillance composites
19-03-2016 6
•Drawn by forensic artist•Based on poor quality surveillance images
![Page 8: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/8.jpg)
May 2, 2023 8
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Example of Verbal Description
The man was about 5ft. 6in. in height, and 34 or 35 years of age, with dark complexion and dark mustache turned up at the ends . . . He had a heavy mustache curled up, and dark eyes and bushy eyebrows. He had no side whiskers, and his chin was clean shaven. He looked like a foreigner."
19-03-2016 7
![Page 9: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/9.jpg)
Objective
SKETCH TO PHOTO MATCHING SYSTEM
Develop a system that can automatically match sketch with mugshots
System that manage every types of sketch such as -forensic sketches -composite sketches -surveillance sketches
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 8
![Page 10: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/10.jpg)
• Law enforcement agencies disseminate a sketch to media outlets
• Hope that someone will recognise the suspect
• And provide information leading to an arrest.
19-02-2016 2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Existing system
19-03-2016 9
![Page 11: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/11.jpg)
May 2, 2023 11
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 10
Existing system – contd…
![Page 12: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/12.jpg)
Problem description
SKETCH TO PHOTO MATCHING SYSTEM
•Slow and tedious process
•Fails to leverage resources available to law enforcement agencies (mugshot databases)
•So there is a need for a method that can automatically match facial composites to large police mugshot databases.
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 11
![Page 13: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/13.jpg)
• Automatically match facial composites against mugshots
• For this 2 algorithms are used• Holistic representation (developed by Klare and Jain)• Component representation (developed by Han et al.)
19-02-2016 2
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Proposed system
19-03-2016 12
![Page 14: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/14.jpg)
May 2, 2023 14
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 13
Proposed system – contd…
![Page 15: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/15.jpg)
SKETCH TO PHOTO MATCHING SYSTEM
Working •Sketches that is given as input to the system
•From sketch features are extracted based on 2 algorithms
•There will be a database to hold the features of large mugshot
•By applying 2 algorithms it matches with the mugshot
•2 algorithms produce matching scores
•Combining matching scores of 2 algorithm
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 14
![Page 16: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/16.jpg)
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
Figure
19-03-2016 15
![Page 17: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/17.jpg)
Holistic algorithm
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 16
•First step is detection of eye
•Alignment and CSDN
•Feature extraction -SIFT -MLBP
•LDA- learn optimal subspaces & projecting into subspaces•PCA
19-03-2016
![Page 18: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/18.jpg)
18
SKETCH EYE DETECTION
ALIGNMENT AND CSDN
PATCHWISE SEGMENTATION
DENSE GRID OF KEYPOINTS
MLBP
SIFT
LDA Patch-wise concatenation & PCA
Sketch Feature Vector
Mugshot Feature Vector
Repeated for all patches
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE
L2 norm
19-03-2016 17
Holistic Algorithm – contd…
![Page 19: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/19.jpg)
Holistic algorithm – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 18
•The same steps are performed for sketch and photo
•Sketch feature vector and mugshot feature vector
•These 2 features are comparing for similarity using L2 norm
•Finding score of matching
19-03-2016
![Page 20: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/20.jpg)
Component-based algorithm
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19
•In holistic algorithm, features are extracted from each patch / grid from images•But component-based algorithm, features are extracted from each component•Per component, features are extracting
19-03-2016
![Page 21: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/21.jpg)
May 2, 2023 21
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 20
Component-based algorithm – contd…
![Page 22: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/22.jpg)
Modules
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 21
Eye Detection
•Accurate and reliable•Eyes are protected and remain reliably unchanged for decades•Eyes are relatively stable feature
19-03-2016
![Page 23: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/23.jpg)
Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2219-02-2016
Alignment and CSDN
•composite is normalized to a fixed height and width•transformed such that right and left eyes are at the same position for every composite.
•The center-surround divisive normalization (CSDN)•filter is then applied to the composite to compensate change in modality between composite and mugshot.
19-03-2016
![Page 24: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/24.jpg)
Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 23
•SIFT and MLBP Feature Extraction
•SIFT features are extracted from dense grid of keypoints
•Multi-scale local binary pattern (MLBP) features are extracted from a uniform patches across the face,
19-03-2016
![Page 25: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/25.jpg)
Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 24
LDA
•For both SIFT and MLBP features, optimal subspaces are learned for each patch using linear discriminant analysis (LDA).•projecting the patch-wise features into their respective subspaces
PCAPCA is applied to the feature vector to reduce template size
19-03-2016
![Page 26: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/26.jpg)
Modules – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 25
L2 norm•It is used as classifier•For finding similarity between mugshot and sketch features
Sum of Score Fusion•Scores from 2 algorithms are fused using sum of score fusion
19-03-2016
![Page 27: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/27.jpg)
Innovative idea
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 26
•Modifying existing systems with Enhanced Holistic method and Enhanced Composite method.
•Replacing feature extraction methods and classifier
19-03-2016
![Page 28: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/28.jpg)
Innovative idea
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2719-03-2016
![Page 29: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/29.jpg)
Innovative idea – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 28
•In the proposed scheme it is done by replacing the existing systems algorithms such as SIFT and MLBP with SURF and harlick algorithms.
•Also adding Neural network classifier instead of L2 norm for better accuracy.
19-03-2016
![Page 30: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/30.jpg)
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2919-03-2016
•Input sketch
![Page 31: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/31.jpg)
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3019-03-2016
•Eye detection from sketch
![Page 32: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/32.jpg)
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3119-03-2016
•Aligned sketch
![Page 33: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/33.jpg)
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3219-03-2016
•Features extracted from sketch
![Page 34: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/34.jpg)
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3319-03-2016
•Eye detection from photo
![Page 35: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/35.jpg)
screenshots – contd…
SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3419-03-2016
•Feature extraction from photo
![Page 36: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/36.jpg)
Conclusion
SKETCH TO PHOTO MATCHING SYSTEM
3619-02-2016
•Facial composites are used by law enforcement agencies for the identification of suspects involved in criminal activities
•Sketch to photo system effectively make the use of facial composites
•Automatic
•Can be used with every type of composites
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 35
![Page 37: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/37.jpg)
Number: 01 Guide Name: Mr. JOBI V DAS, AP/CSE 36
SKETCH TO PHOTO MATCHING SYSTEM
•For that 2 algorithms are using here
•Holistic algorithm and component based algorithm
•Match scores from 2 algorithms are combining
•Also filtering based on demographic information
•Along with the matched image some details such as name, age can be displayed.
19-3-2016
Conclusion – contd…
![Page 38: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/38.jpg)
References SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 3819-02-2016
•Bonnen K. , B. F. Klare, and A. K. Jain, “Component-based representation in automated face recognition,” IEEE Trans. Inf. Forensics Security, vol. 8, no. 1, pp. 239–253, Jan. 2012.
•Best-Rowden L. , H. Han, C. Otto, B. Klare, and A. Jain, “Unconstrained face recognition: Identifying a person of interest from a media collection,” IEEE Trans. Inf. Forensics Security,Dec. 2014
•Han H. , B. F. Klare, K. Bonnen, and A. K. Jain, “Matching composite sketches to face photos: A component-based approach,” IEEE Trans. Inf. Forensics Security, vol. 8, no. 1, pp. 191–204, Jan. 2013
19-03-2016
![Page 39: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/39.jpg)
References – Contd… SKETCH TO PHOTO MATCHING SYSTEM
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 39
•Scott J. Klum, Hu Han, Brendan F. Klare Anil K. Jain, ”The FaceSketchID System: Matching Facial Composites to Mugshots” IEEE Trans. Inf. Forensics Security,Dec. 2014 ,Vol. 9, no. 12, Dec 2014
19-03-2016
![Page 40: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/40.jpg)
Quires
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 40
SKETCH TO PHOTO MATCHING SYSTEM
May 2, 2023
?
![Page 41: sketch to photo matching](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e73d361a28ab8f028b56a1/html5/thumbnails/41.jpg)
Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 41
SKETCH TO PHOTO MATCHING SYSTEM
May 2, 2023
Thank you