Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss...
Transcript of Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss...
![Page 1: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/1.jpg)
Machine Learning and Safety in
Automotive SoftwareRick Salay
February 4, 2019
![Page 2: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/2.jpg)
© 2018 Rick Salay 2
![Page 3: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/3.jpg)
Agenda – two strategies for safety assurance of ADS and ML
1. Hazard-based automotive safety standard (ISO 26262)
• Will focus on key ML obstacles to V&V
– lack of specification
– lack of interpretability
• Will discuss research directions to address these
2. Measurement uncertainty-reduction based (specifically for
perception)
• Identifying factors contributing to uncertainty and methods to address
them
© 2018 Rick Salay 3
![Page 4: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/4.jpg)
Safety Argument Decomposition
4
PerceptionPlanning &
control
ADS
Sensing ActuationWorld model
Focus on Perception and Supervised Learning
![Page 5: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/5.jpg)
Two ways to implement software: Programming vs. Training (ML)
Programming
Specifications
Training
Examples
© 2018 Rick Salay 5
![Page 6: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/6.jpg)
Safety through a hazard-based automotive safety standard (ISO 26262)
© 2018 Rick Salay 6
![Page 7: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/7.jpg)
Safety Lifecycle (ISO 26262)
(software focus)
© 2018 Rick Salay 7
Where ML is used
![Page 8: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/8.jpg)
© 2018 Rick Salay 8
Select model type
Collect dataset
Select model hyper-parameters
Train model (training dataset)
Evaluate generalization
Test model(testing dataset)
Iterate to optimize hyper-parameters
for over-fitting
Iterate to optimize error rate
Unit design & implementation Unit testing
ML lifecycle
for supervised
learning
![Page 9: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/9.jpg)
ISO 26262 Approach to Software Safety
Assumption: following the recommendations reduces residual risk of hazard
due to SW failure to an acceptable level
Recommends a particular level of rigor in developing safety critical software - different levels for ASIL A-D
- consists of 83 software development techniques (34 at unit level)
© 2018 Rick Salay 9
![Page 10: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/10.jpg)
Best Practices
© 2018 Rick Salay 10
![Page 11: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/11.jpg)
Verification
© 2018 Rick Salay 11
![Page 12: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/12.jpg)
Testing
© 2018 Rick Salay 12
![Page 13: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/13.jpg)
Fault Tolerance
© 2018 Rick Salay 13
![Page 14: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/14.jpg)
Best Practices Prevent faults
Verification Find and repair faults
(and build confidence)Testing
Fault Tolerance Live with faults
Techniques
Unit
Level
© 2018 Rick Salay 14
Assumes programmed software!
![Page 15: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/15.jpg)
Best Practices Prevent faults
Verification Find and repair faults
(and build confidence)Testing
Fault Tolerance Live with faults
Q: How well do ISO 26262 software
recommendations apply to ML components?
Techniques
© 2018 Rick Salay 15
Based on: Salay, Rick, and Krzysztof Czarnecki. "Using machine learning safely in automotive software:
An assessment and adaption of software process requirements in iso 26262." arXiv preprint
arXiv:1808.01614 (2018).
![Page 16: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/16.jpg)
Software technique classification
N/A – technique is not applicable to ML
Adapt – technique can be applied to ML with some adaptation
Use – technique can be used with ML as-is
© 2018 Rick Salay 16
![Page 17: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/17.jpg)
Techniques
Best Practices Prevent faults
VerificationFind and repair faults
Testing
Fault Tolerance Live with faults
© 2018 Rick Salay 17
![Page 18: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/18.jpg)
Best Practices
Strongly biased toward (imperative) programming languages!
Consist of coding guidelines, notation styles, principles
Mostly
N/A
© 2018 Rick Salay 18
![Page 19: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/19.jpg)
What about ML-specific best practices?
Best practices are emerging
E.g. standardized methods for deep neural networks
ML has low maturity compared to traditional programming
© 2018 Rick Salay 19
![Page 20: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/20.jpg)
Techniques
Best Practices Prevent faults
VerificationFind and repair faults
Testing
Fault Tolerance Live with faults
© 2018 Rick Salay 20
![Page 21: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/21.jpg)
“V” Model of Software Development
Specification of software
safety requirements
Software architectural
design
Software unit design and
implementation
Verification of software
safety requirements
Software integration and
testing
Software unit testing
Design phaseverification
Design phaseverification
Software
testing
Software
testing
Software
testing
© 2018 Rick Salay 21
![Page 22: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/22.jpg)
Techniques that are adaptable to ML
Adapt: Static analysis of trained models is feasible
e.g., NN property checking via SMT
© 2018 Rick Salay 23
![Page 23: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/23.jpg)
Techniques directly applicable to ML
Use: Black box testing can be done on ML
components
© 2018 Rick Salay 24
![Page 24: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/24.jpg)
Training
Programming
Specifications
Examples
“V” model assumes that complete specifications exist and are
sufficiently detailed
No specification,
only training dataset
• Dataset always
incomplete
Complete Specification Assumption
© 2018 Rick Salay 25
![Page 25: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/25.jpg)
Training
Programming
Specifications
Examples
Many verification and testing techniques require that the implementation be
human understandable (interpretable)
Written by
humans
Not interpretable
Interpretability Assumption
© 2018 Rick Salay 26
![Page 26: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/26.jpg)
Impact of specification and interpretability on verification and
testing techniques (unit level)
Verification Testing
No specification
Interpretable0.50 (0.0)
0.52(0.03)
Specification
Not interpretable0.26(0.01)
0.97(0.01)
Mean (Std dev) across ASILs
Perfect score is 1.0
Summary
• Specification is important for verification and testing
• Interpretability is critical for verification
© 2018 Rick Salay 28
![Page 27: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/27.jpg)
Complete Specification Assumption
Is the complete specification assumption reasonable?
Not for advanced functionality: ADAS, ADS
Hard to specify:
Perception tasks
e.g., What are complete
necessary and sufficient
conditions to identify a
pedestrian?
Hard to specify:
Planning tasks
in an open
environment
© 2018 Rick Salay 29
![Page 28: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/28.jpg)
Is the complete specification assumption reasonable?
Not for advanced functionality: ADAS, ADS!
No specification => hard to direct a programmer
Conclusion: Machine Learning is preferred approach!
No specification => nothing to verify against!
Complete Specification Assumption
© 2018 Rick Salay 30
![Page 29: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/29.jpg)
Some specifications with ML components still possible: two kinds
Partial behavioural specifications
(PBS)
Assumptions
e.g. illumination > 15000 lux
Necessary/Sufficient conditions
e.g., pedestrian < 9 feet tall
Invariants, equivariants
e.g., classification is invariant to rotation
Complete Specification Assumption : How to address?
© 2018 Rick Salay 31
Complete data specifications
(DS)Domain coverage requirements
e.g., pedestrian equivalence
classes
Risk profiling of inputse.g. severity of misclassifying
different subclasses of objects
![Page 30: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/30.jpg)
© 2018 Rick Salay 32
Select model type
Collect dataset
Select model hyper-parameters
Train model (training dataset)
Evaluate generalization
Test model(testing dataset)
Iterate to optimize hyper-parameters
for over-fitting
Iterate to optimize error rate
Unit design & implementation Unit test
Where to Use Specifications
![Page 31: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/31.jpg)
© 2018 Rick Salay 33
Select model type
Collect dataset
Select model hyper-parameters
Train model (training dataset)
Evaluate generalization
Test model(testing dataset)
Iterate to optimize hyper-parameters
for over-fitting
Iterate to optimize error rate
Unit design & implementation Unit test
Select model type that implicitly satisfies
PBS
• e.g., CNN’s satisfying required
equivariants• Cohen T, M. Welling, 2016. “Group equivariant
convolutional networks.” In International Conference on
Machine Learning (pp. 2990-2999).
Where to Use Specifications
![Page 32: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/32.jpg)
© 2018 Rick Salay 34
Select model type
Collect dataset
Select model hyper-parameters
Train model (training dataset)
Evaluate generalization
Test model(testing dataset)
Iterate to optimize hyper-parameters
for over-fitting
Iterate to optimize error rate
Unit design & implementation Unit testUse active learning to accelerate optimization by
selecting the most informative dataE.g., Sivaraman, S., M. M. Trivedi. 2014. “Active learning for on-road vehicle
detection: A comparative study”. Machine vision and applications: 1-13.
Where to Use Specifications
![Page 33: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/33.jpg)
© 2018 Rick Salay 35
Select model type
Collect dataset
Select model hyper-parameters
Train model (training dataset)
Evaluate generalization
Test model(testing dataset)
Iterate to optimize hyper-parameters
for over-fitting
Iterate to optimize error rate
Unit design & implementation Unit test
• Ensure dataset satisfies specs
• DS (e.g., all ped poses represented)
• PBS (e.g., all ped <= 9ft)
• Use PBS to augment dataset so that specs
are learned by model
• e.g., generate non-ped > 9ft
• e.g., use GANs to generate invariant
examples• Liu M.Y., T. Breuel, J. Kautz, 2017. “Unsupervised
Image-to-Image Translation Networks”. arXiv preprint
arXiv:1703.00848
Where to Use Specifications
![Page 34: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/34.jpg)
© 2018 Rick Salay 36
Select model type
Collect dataset
Select model hyper-parameters
Train model (training dataset)
Evaluate generalization
Test model(testing dataset)
Iterate to optimize hyper-parameters
for over-fitting
Iterate to optimize error rate
Unit design & implementation Unit test
Incorporate PBS into loss function to
penalize models that violate them and guide
learning.
e.g., Xu J., Z. Zhang, T. Friedman, Y. Liang, G.V. Broeck, 2017. “A
Semantic Loss Function for Deep Learning with Symbolic
Knowledge”. arXiv preprint arXiv:1711.11157.
Vedaldi A., M. Blaschko, A. Zisserman, 2011. “Learning equivariant
structured output SVM regressors.” In Computer Vision (ICCV),
IEEE International Conference on 2011 (pp. 959-966). IEEE.
Where to Use Specifications
![Page 35: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/35.jpg)
© 2018 Rick Salay 37
Select model type
Collect dataset
Select model hyper-parameters
Train model (training dataset)
Evaluate generalization
Test model(testing dataset)
Iterate to optimize hyper-parameters
for over-fitting
Iterate to optimize error rate
Unit design & implementation Unit test
• Use test coverage metrics
designed for ML• E.g., Sun, Y., X. Huang, and D. Kroening.
"Testing Deep Neural Networks." arXiv preprint
arXiv:1803.04792 (2018).
• Use explanation techniques to
diagnose why tests pass or fail• Koopman, P. and M. Wagner. "Toward a
Framework for Highly Automated Vehicle
Safety Validation," SAE World Congress,
2018. SAE-2018-01-1071.
Where to Use Specifications
![Page 36: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/36.jpg)
© 2018 Rick Salay 38
Select model type
Collect dataset
Select model hyper-parameters
Train model (training dataset)
Evaluate generalization
Test model(testing dataset)
Iterate to optimize hyper-parameters
for over-fitting
Iterate to optimize error rate
Unit design & implementation Unit test
Verify trained model wrt PBS
- multiple techniques
Where to Use Specifications
![Page 37: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/37.jpg)
Verification Techniques for ML
© 2018 Rick Salay 39
Requires interpretability – use interpretability enhancing
techniques discussed below
Where to Use Specifications
![Page 38: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/38.jpg)
Verification Techniques for ML
© 2018 Rick Salay 40
Combine formal and non-formal techniques
e.g. falsification• Dreossi, T., A. Donzé, and S.A. Seshia. "Compositional falsification of cyber-physical systems
with machine learning components." In NASA Formal Methods Symposium, pp. 357-372.
Springer, Cham, 2017.
Where to Use Specifications
![Page 39: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/39.jpg)
Verification Techniques for ML
© 2018 Rick Salay 41
Proof that model satisfies PBS• Seshia, S.A., D. Sadigh, and S.S. Sastry. "Towards verified artificial intelligence." arXiv
preprint arXiv:1606.08514 (2016).
Proof of minimum adversarial attack radius
Where to Use Specifications
![Page 40: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/40.jpg)
Verification Techniques for ML
© 2018 Rick Salay 42
These are code-specific techniques.
Where to Use Specifications
![Page 41: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/41.jpg)
Verification Techniques for ML
© 2018 Rick Salay 43
PBS property checking• E.g., Katz, G., C. Barrett, D. Dill, K. Julian, and M. Kochenderfer. 2017. “Re-luplex: An
Efficient SMT Solver for Verifying Deep Neural Networks". arXiv preprint arXiv:1702.01135
Abstract Interpretation• E.g., Gehr, T., M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev.
"AI2: Safety and robustness certification of neural networks with abstract interpretation." In
Security and Privacy (SP), 2018 IEEE Symposium on. 2018.
Where to Use Specifications
![Page 42: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/42.jpg)
Verification Techniques for ML
© 2018 Rick Salay 44
Translate the model to another semantically equivalent
representation for which analysis tools exist• E.g., Weiss, G., Y. Goldberg, and E. Yahav. "Extracting Automata from Recurrent Neural
Networks Using Queries and Counterexamples." arXiv preprint arXiv:1711.09576 (2017).
Where to Use Specifications
![Page 43: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/43.jpg)
More powerful ML model => Less
interpretableaccuracy/interpretability tradeoffNaïve
Bayes
Support
Vector
Machine
Random
Decision
Forest
Deep
Neural
Network
Interpretability Assumption
© 2018 Rick Salay
Decision
TreeBayesian
Network
45
![Page 44: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/44.jpg)
Require use of interpretable modelsor, provide justification why not (safety case)
and use interpretability increasing techniques
Natural Language Rule Extraction
Interpretability Assumption : How to address?
© 2018 Rick Salay 46
Model Visualization Dependency Analysis
Saliency Maps DARPA XAI
![Page 45: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/45.jpg)
Global visualization: t-SNE*
through dimensionality
reduction MNIST in 2D
t-SNE
(t-distributed
Stochastic
Neighbor
Embedding)
© 2018 Rick Salay 47
* Maaten, L. van der, and G. Hinton. "Visualizing data using t-SNE." Journal of
machine learning research no. 9, Nov (2008): 2579-2605.
Interpretability Increasing Techniques
![Page 46: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/46.jpg)
MNIST *
data set
* http://yann.lecun.com/exdb/mnist/
© 2018 Rick Salay 48
![Page 47: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/47.jpg)
Global visualization: Activation Maximization
© 2018 Rick Salay 49
data: MNIST
Layer 1 Layer 2 Layer 3
* Erhan, Dumitru, Yoshua Bengio, Aaron Courville, and Pascal Vincent. "Visualizing
higher-layer features of a deep network." University of Montreal 1341, no. 3 (2009): 1.
Interpretability Increasing Techniques
![Page 48: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/48.jpg)
Global feature importance
© 2018 Rick Salay 50
data: MNIST
https://www.kaggle.com/c/digit-recognizer/discussion/70350
Interpretability Increasing Techniques
![Page 49: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/49.jpg)
Local explanation: Occlusion map
© 2018 Rick Salay 51
data: MNIST
Interpretability Increasing Techniques
![Page 50: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/50.jpg)
Techniques
Best Practices Prevent faults
VerificationFind and repair faults
Testing
Fault Tolerance Live with faults
© 2018 Rick Salay 53
Summary
• Key assumptions are not met by ML: complete specification and
interpretability
• However, research is active in these areas to address the
shortfall
![Page 51: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/51.jpg)
Techniques
Best Practices Prevent faults
VerificationFind and repair faults
Testing
Fault Tolerance Live with faults
© 2018 Rick Salay 54
![Page 52: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/52.jpg)
Fault Tolerance
Can use as-is for ML
Fault tolerance strategies are architecture-level and can be
component implementation agnostic
But error detection/handling should use programming!
© 2018 Rick Salay 55
![Page 53: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/53.jpg)
Some ML-oriented Fault Tolerance Methods
© 2018 Rick Salay 56
Ensemble methodsUse multiple classifiers and aggregate their results
Safety envelopeUse ML components only within safe contexts – e.g. to choose among a set of safe actions
Simplex architectureMonitor when ML component is unreliable and switch to a reliable (but usually conservative)
non-ML component – requires “uncertainty” check on ML component
Runtime verification + Fail SafetyMonitor PBS satisfaction and go to fail safe behaviour if PBS is violated at run-time
![Page 54: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/54.jpg)
Summary
Best Practices Prevent faults
VerificationFind and
repair faultsTesting
Fault
Tolerance
Live with
faults
Q: How well do ISO 26262 SW recommendations fit ML?
N/A – but ML best practices will
emerge (unclear of impact)
Adapt/Use – if specification and
interpretability problems are
addressed (research is active)
Use – Fault tolerance techniques
can be used directly
© 2018 Rick Salay 57
![Page 55: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/55.jpg)
What about planning & control?
58
PerceptionPlanning &
control
ADS
Sensing ActuationWorld model
![Page 56: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/56.jpg)
Main type of ML in actuation/control: Reinforcement Learning (RL)
• learn an optimal control policy by training with simulation + reward
function
• exploration/exploitation trade-off
• leaning could be on-line as well
Some unique safety issues:
• e.g., reward function does not incorporate (safety) risk
• e.g., model learns to “game” the reward function
• See: Amodei, D., C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D.
Mane. 2016. “Concrete problems in AI safety". arXiv preprint arXiv:1606.06565 .
© 2018 Rick Salay 59
![Page 57: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/57.jpg)
Safety through (Measurement) Uncertainty-Reduction
© 2018 Rick Salay 60
![Page 58: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/58.jpg)
Managing Perceptual Uncertainty in ML
61
PerceptionPlanning &
controlGuaranteedperceptionperformance
ODDassumptions
Guaranteedplanning &controlperformance
ADS
Sensing ActuationWorld model
Actuator + ODDAssumptions
The following slides are based on Krzysztof Czarnecki and Rick Salay.Towards a Framework to Manage Perceptual Uncertainty for Safe Automated Driving.In WAISE, Västerås, Sweden, 2018https://uwaterloo.ca/wise-lab/publications/towards-framework-manage-perceptual-uncertainty-safe
![Page 59: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/59.jpg)
Guide to the Expression of Uncertainty in Measurement (GUM)
• True accuracy unknowable
– Accuracy in ML wrt. test set only
• Must estimate uncertainty
62
![Page 60: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/60.jpg)
Sample Scenario-DependentPerception-PerformanceSafety-Requirement Spec
63
Detect pedestrians on the roadwaywithin range of 10 m and with maximum perception-reaction delay of 0.5 swith missed detection probability of 10-9 or lesswith localization uncertainty of ± 0.5 m or betterwithin ODD conditions
Detection range
Stopping sight distanceStopping
buffer
![Page 61: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/61.jpg)
Perception Triangle (Instance-Level)
64
Perception
Real-world situation
Sensorychannel
Cameraimage,radardataPerception
algorithm
Pedestrianspeed = 0.1activity =
walking
Pedestrianspeed = 0activity =
standing
…
Set of credible states(uncertain)
Accuracy
Pedestrianspeed = 0activity =
standing
True state(unknowable)
![Page 62: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/62.jpg)
Perceptual Triangle
65
Real-world situations
Concept
Semantics
Sensorydata
Sensorychannel
Datainterpretation
Perception
Sensorychannel
Perception
Real-world situation
Pedestrianspeed = 0activity =
standing
True state(unknowable)
Perceptionalgorithm
Cameraimage,radardata
Pedestrianspeed = 0.1activity =
walking
Pedestrianspeed = 0activity =
standing
…
Set of credible states(uncertain)
Accuracy
Instance-level Domain-level (generic)
![Page 63: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/63.jpg)
Perceptual Triangle When Using Supervised ML
66
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Training& testing
TrainedModel
Modelclass selection,training & testing
Development
Inference
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
Operation
![Page 64: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/64.jpg)
Factors Influencing Uncertainty
67
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
Training& testing
Inference
TrainedModel
Modelclass selection,training & testing
Development Operation
![Page 65: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/65.jpg)
F1: Conceptual Uncertainty
68
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
Training& testing
Inference
TrainedModel
Modelclass selection,training & testing
Development Operation
F1
![Page 66: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/66.jpg)
F1: Conceptual UncertaintyPedestrian or Cyclist?
69
![Page 67: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/67.jpg)
F1: Conceptual Uncertainty
• Assessed by expert review or labeling disagreement
• Reduced by developing standard ontologies
– E.g., WISE Drive Ontology
70https://uwaterloo.ca/wise-lab/projects/wise-drive-requirements-analysis-framework-automated-driving
![Page 68: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/68.jpg)
F2: Development Scenario Coverage
71
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
Training& testing
Inference
TrainedModel
Modelclass selection,training & testing
Development Operation
F1
F2
![Page 69: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/69.jpg)
F2: Development Scenario Coverage
72
![Page 70: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/70.jpg)
F2: Development Scenario Coverage
• Assessed with respect to ontologies and field validation targets• Must include positive/negative and near-hit/near-miss examples
• Challenge: how much data is enough?
73
![Page 71: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/71.jpg)
Synthetic data sets
74
Angus et al. Unlimited Road-scene Synthetic Annotation (URSA) Dataset, ITCS’18
https://uwaterloo.ca/wise-lab/ursa
![Page 72: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/72.jpg)
Active Learning
Data selection criteria
1. Uncertainty
2. Coverage & diversity
3. Collection & labeling cost
4. Risk profile
75
![Page 73: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/73.jpg)
F2
F3: Scene Uncertainty
76
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
Training& testing
Inference
TrainedModel
Modelclass selection,training & testing
Development Operation
F1
F3
![Page 74: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/74.jpg)
F3: Scene Uncertainty
77
![Page 75: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/75.jpg)
F3: Scene Uncertainty
• Surrogate measures
– range, scale, occlusion level, atmospheric visibility, illumination, clutter and crowding level
• May compare test set accuracy and output confidence with these measures
• Also part of development data set coverage
78
![Page 76: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/76.jpg)
F3F2
F4: Sensor Properties
79
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
Training& testing
Inference
TrainedModel
Modelclass selection,training & testing
Development Operation
F1F4
![Page 77: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/77.jpg)
F4: Sensor Properties
80
![Page 78: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/78.jpg)
F4: Sensor Properties
• Mature engineering discipline
– Determining sensor properties to capture sufficient information
– Mode, range, resolution, sensitivity, placement, etc.
• However, interaction between ML algorithms and sensor properties must be assessed
– E.g., how effective is ML is ignoring sensor noise or artifacts?
81
![Page 79: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/79.jpg)
F3F2
F5: Label Uncertainty
82
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
Training& testing
Inference
TrainedModel
Modelclass selection,training & testing
Development Operation
F1F4
F5
![Page 80: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/80.jpg)
F5: Label Uncertainty
83
Class: cyclist vs. pedestrian Bounding box placement uncertainty
3D bounding box placement is challenging
![Page 81: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/81.jpg)
F5: Label Uncertainty
• Assessed by expert review and labeler disagreement– Existing research on determining number of labelers in crowd
sourcing
– E.g., may need as many as 6 independent votes
• Reduction measures– Conceptual clarity (F1)
– Quality control• Clear instructions, training, verification, etc.
• Bread and butter of labeling companies
84
![Page 82: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/82.jpg)
F3F2
F6: Model Uncertainty
85
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
Training& testing
Inference
TrainedModel
Modelclass selection,training & testing
Development Operation
F1F4
F5
F6
![Page 83: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/83.jpg)
F6: Model Uncertainty
86
What model was learned in training?What decisions will it make in operation?
![Page 84: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/84.jpg)
F6: Model Uncertainty
1. Explanation methods help validate features
2. Robustness measures help assess risk of misclassification
3. Bayesian deep learning can help assess model uncertainty
87
![Page 85: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/85.jpg)
Deep Learning and Explanations
88
Passenger car
The top 15 features (superpixels) used to classify corresponding input imageas a car by an Inception network trained on ImageNet
The explanation showsthat a tree contributedto the classificationdecision(method: LIME)
(see LIME at https://github.com/marcotcr/lime)
![Page 86: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/86.jpg)
Adversarial Stickers
89Evtimov et al.
Misclassified as speed signs
![Page 87: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/87.jpg)
Robustness Measures
90CLEVER approach by IBM
![Page 88: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/88.jpg)
Aleatoric and Epistemic Uncertainty
91Yarin Gal, et al., https://arxiv.org/abs/1703.04977
![Page 89: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/89.jpg)
Dropout
92
![Page 90: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/90.jpg)
Methods for Confidence Estimation
1. Model uncertainty using MC Dropout
2. Data uncertainty using heteroschedastic regression
3. Confidence calibration
93
Phan, Salay, Czarnecki, Abdelzad, Denouden, Venekar.Calibrating Uncertainties in Object Localization Task.NIPS workshop. 2018, https://arxiv.org/abs/1811.11210 95% confidence band
Predicted mean box
Ground truth
![Page 91: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/91.jpg)
F7: Operational Domain Uncertainty
Concept
Developmentsituations and
scenarios
Sensorydata
Sensorychannel
Partialsemantics(examples)
Datalabeling
Concept
Operationalsituations and
scenarios
Sensorydata
Sensorychannel
Resultingperception
Inferredstate
F4Training& testing
Inference
TrainedModel
Modelclass selection,training & testing F6
F5
Domain shift
Development Operation
F2 F3
F1
94
F7
F2
F4
F3
![Page 92: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/92.jpg)
F7: Operational Domain Uncertainty
95Camera miscalibration
Fly splatters on LIDAR
New type of car shape
New pedestrian pose
F2 F3
F4
F4
![Page 93: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/93.jpg)
F7: Operational Domain Uncertainty
• Assess situation novelty at operation time
– E.g., autoencoders, partial specs
• Assess impact of level of sensor miscalibration on perceptual uncertainty
• Monitor sensor parameters and ODD
96
![Page 94: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/94.jpg)
Sample Incorrect Detections
97
scoreIoU
missed
![Page 95: Machine Learning and Safety in Automotive Softwarechechik/courses19/csc2125/... · Semantic Loss Function for Deep Learning with Symbolic Knowledge”. arXiv preprint arXiv:1711.11157.](https://reader033.fdocuments.us/reader033/viewer/2022050502/5f94e117fb9aaa4933518405/html5/thumbnails/95.jpg)
Thank you
Questions?