Post on 26-Jun-2020
Electronic board defect classification and detection with deep learning
Dan Sebban – VP of Data Analysis, Optimal+
Nissim Matatov – Machine Learning Engineer, Optimal+
What is machine learning
Machine learning is a field of computer science that gives computers the ability to learn without being explicitly programmed
Machine learning and AI
Artificial intelligence
Machine learning
Deeplearning
Artificial intelligence: Any technique that enables computers to mimic human intelligence.
Machine learning: A subset of AI that includes statistical techniques that enable machines to improve at tasks with experience.
Deep learning: The subset of machine learning that permits software to train itself to perform tasks like speech and image recognition, by exposing neural networks to vast amounts of data
Artificial intelligence (AI)
enables computers to mimic human intelligence by providing key human abilities
Bunny is in picture
Computer visionkey AI capability
Deep learningkey methodology to achieve AI Bunny is here
Image classification: Who is in picture?
Seecomputervision
Hearspeech recognition
Comprehendnatural language processing
Deep learning in electronics –MotivationVisual Inspection (VI) aims to check for the presence of defects on a board
Before: Manual visual inspection
• Very laborious – impacts operational and manufacturing efficiency
• Depends on human ability to recognize defects – impacts quality
After: Computer-aided visual inspection
• Automated – improves efficiency
• More accurate in recognizing defects –improves quality
Case study introduction
• Electronic Ceramic Substrate welded through several pins to a housing
• All parts are inspected as part of the manufacturing flow using a Surface Acoustic Microscope (SAM)
• Images are generated by the microscope
• Images are labeled by technician as “Defect”/”No Defect”
Defect Image ClassificationWhether an image contains a defect
Computer vision tasks in electronics
Score : 0.08 Score : 0.32 Score : 0.92
Score for “Star” crack:
Score : 0.08 Score : 0.18 Score : 0.97
Defect ClassificationWhether an image contains a specific defect type (e.g. crack with “Star” shape)→ Allows root cause analysis
Defect DetectionWhere the defect is actually located
Business goalMaximum test time saving through VI reduction at minimum quality impact
ML goal Defect Image Classification and accurate prediction of boards without defects
ML task Supervised ML for binary classification (Defect / No Defect)
Action to save test time
Skip boards which are predicted safe (No Defect) with high degree of confidence
Evaluation Estimated number of boards with undetected defects per VI reduction level
Deep learning framing
Typical VI
inspection flow
Image Capture by
SAM
Manual InspectionPass/Fail
Proceed to next test
No Defect
Deep learning
model is
deployed to a
factory floor
Image Capture by
SAM
DL modelScore each board image for defect
Proceed to next
operation
Manual Inspection
High scored boards
Low scored boards
Defect
Engineering Disposition
Deep learning model deployment
Input data DL evaluationDL modeling
• “Defect”/”No Defect” ratio: ~1:10
• Manual labeling “Defect”/”No Defect” of 600 images
• VGG-16 Convolution Neural Network (CNN) structure
• Image augmentation pre processing procedure
• Transfer Learning
• Hyperparameter optimization for CNN
• Ensembling
• Modeling data (600 images): “Defect”/”No Defect” ratio: 1:5
• Evaluation(300 images*10 iterations): “Defect”/”No Defect” ratio: 1:10
Deep learning model deployment
No Defect Defect
No
Defect
TP(True positive)
FN(False negative)
Defect
FP(False positive)
TN(True negative)
Predicted
Classic classification performance measuresTP + TNTP + TN + FP + FNAccuracy =
Evaluation data: 12 boards (8 “No Defect” + 4 “Defect”)Random board selection → 50% skip = 2 escapes
Model based board selection (based on scoring) → 50% skip = 1 escape
Lift = 2 [DL model is 2x better than random selection]50% selection
TNTN + FPRecall(Sensitivity)
=
Lift [more intuitive metric] = 𝐑𝐚𝐧𝐝𝐨𝐦 𝐒𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧𝐌𝐨𝐝𝐞𝐥 𝐒𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧
AUC-ROC =
Example
Technical evaluation
Business evaluation
Skip level vs. avg. escapes tradeoff
Proposed skip level = 40%
60% of the boards will continue with human VI
40% of the boards will skip VI
Escape rate is 1%, or ~ 3 boards out of 300
Escapes ratio becomes faster for higher skip levels
Escape ratio is acceptable by customer for potential TTR
Strong DL model requires one time effort to create accurate
“Defect”/”No Defect” labeling
Existing image labeling isn’t totally accurate and negatively impacts model performance
Skip level 40%
VI escaped boards - Random (boards) 12.5
VI escaped boards - Random (% of total) 4.17%
VI escaped boards - Model (boards) 2.9
VI escaped boards - Model (% of total) 0.97%
Lift 4.3
Examples for Prediction vs. Actual
Case 1
Prediction = “No Defect” and Actual = “No Defect”The model is confident that defect is present despite poor image quality
Case 2
Prediction = “No Defect” and Actual = “Defect”Room for model improvement
Case 3
Prediction = “Defect” and Actual = “No Defect” Defect is clearly seen: labeling should be revised to improve the model
Defect detection evaluation
IoU = 0.79 IoU = 0.18
Actual bounding box
Predicted bounding boxv
Evaluation metric:
IoU (Intersection over Union)
DL model with mean IoU > 0.5 is considered strong
Case study mean IoU metric is ~0.65
Deep neural network in a nutshell
Input Layer
Pixelwise image presentation
Convolution + Pooling layers
Learn about elements of image , i.e. edges
of objects
Full Connected Layer
Learn about object presence from previous
information
Classification Layer
Express confidence about object presence
Deep learning methodologies
Data augmentation
Transform original images to create new images for learning
Transfer learning
Use previously accumulated knowledge during the learning
External feature embedding
Use other inspections along with information learned from image
Assembling
Combine partial models to provide more accurate final prediction
DL visualization
Watch that intermediate results make sense
The machine learning process
Adapt
Learn Act
Validate
Learn from data and evaluate business value
Understand changes and update model
Deploy and act upon the model
Monitor model performance and identify changes
ML/DL application challenges
Fully automated test process
High cost to interrupt the inspection process
Low volume - Low cost products
Effort on adaptivity is ineffective
New product or new technology
Not enough relevant images are available for analysis
Qualitative data
Images quality and their correct labeling
High performance DL
Computationally expensive
Model deployment and operationalizing
Model benefit is materialized and maintained
ML/DL deployment ecosystem
The algorithm is not enough – it’s the infrastructure that is challenging
Google article from 2014: Hidden Technical Debt in Machine Learning Systems
Key takeaways
• Significant value was demonstrated using Deep Learning for Defects Inspection
• Such methodology improves both operational efficiency, manufacturing throughput,
overall product quality, and reliability
• Deep learning is independent of defect types and locations, and therefore a more
suitable methodology than classical image processing on some specific tests cases
• The recent significant improvements in Deep Learning techniques allows to get faster
and more reliable models
Thank you!