Post on 28-Dec-2015
The Fish4Knowledge Project
Disclosing Computer Vision Errors to End-Users
Emma Beauxis-Aussalet, Lynda Hardman, Jacco Van Ossenbruggen, Jiyin He,Elvira Arslanova, Tiziano Perrucci
12 December 2014 CWI Scientific Meeting 1
Monitoring Fish Population
3
Why use Computer Vision?• It can count fish from each species• It supports short- to long-term research• It is not intrusive, and cost-effective
Monitoring Fish Population
4
But… new practices need to be introduced,and scientific validity needs to be assessed.
Why use Computer Vision?• It can count fish from each species• It supports short- to long-term research• It is not intrusive, and cost-effective
7
Collect examples of fish(Ground-Truth)
Construct fish models
Classify fish species as the most similar model
Detecting Fish
Motivations for HCI Research
8
Support uncertainty-aware data analysis • What are the uncertainty factors?• How to inform ecologists about each factor?• How to support user assessment of end-results?
Here the Octopus appeared.
(½Φ )-(π√⅞)
How precise is this?
Interactions of Uncertainty Factors
12
Poor ground-truth yields poor models
Ground-Truth Quality
Computer Vision Errors
Interactions of Uncertainty Factors
13
Poor imagesyield more errors
Ground-Truth Quality
Computer Vision Errors
Image Quality
Interactions of Uncertainty Factors
14
Typhoons yield poor images?What confidence intervals?
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
15
Missing videos?
Number of Video Samples
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
16
Some species swim in & out the field of view
Number of Video Samples
Duplicated Individuals
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
17
Fields of view target specific species
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
18
Fields of view target specific species
and shift overtimeNumber of Video
Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
19
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Biases & Noisein Specific Output
Image Quality
Conveying Uncertainty Factors
20
Confusion Matrices
Computer Vision Errors
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Image Quality
Conveying Uncertainty Factors
21
Confusion Matrices
Computer Vision Errors
Biases & Noisein Specific Output
LogisticRegression
12 December 2014 CWI Scientific Meeting
Conveying Computer Vision Errors with Confusion Matrices
22
Here the Octopus appeared.
(½Φ )-(π√⅞)
How precise is this?
12 December 2014 CWI Scientific Meeting
Conveying Computer Vision Errors with Confusion Matrices
23
Here the Octopus appeared.
(½Φ )-(π√⅞)
How precise is this?
Without torture, no science.
Russian Proverb
Proposed Metric & Visualization
33
• 3 basic concepts: correct, added, missed fish• 2 Main sources of errors• Number & Proportions of error • Simple metric for extrapolations
Proposed Metric & Visualization
34
• 3 basic concepts: correct, added, missed fish• 2 Main sources of errors• Number & Proportions of error • Simple metric for extrapolations
40
Issues Tackled
Large number of TNconceals uncertainty
Information is lostabout errors interdependence
41
Issues Tackled
Information is lostabout errors interdependence
FP for one class are FN for another
Large number of TNconceals uncertainty
42
Issues Tackled
Information is lostabout errors interdependence
Large number of TNconceals uncertainty
Class proportions can vary
43
Issues Tackled
Information is lostabout errors interdependence
Large number of TNconceals uncertainty
Class proportions can vary
Using one single type of curvecan hide differences
12 December 2014 CWI Scientific Meeting
ConveyingComputer Vision Biaseswith Logistic Regression
44
in collaboration with Bas Boom
Without torture, no science.
Russian Proverb
Achievements & Limitations
But beware further variations of class proportions!
Before After
Biases arereduced
Fish counts are improved
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Image Quality
Future Work
58
Computer Vision Errors
Biases & Noisein Specific Output
Integrate Computer Vision errorsinto ecologists’ statistical framework
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Image Quality
Future Work
59
Computer Vision Errors
Biases & Noisein Specific Output
Integrate Computer Vision errorsinto ecologists’ statistical framework
User studies with our visualizations
Number of Video Samples
Ground-Truth Quality
Image Quality
Future Work
60
Integrate Computer Vision errorsinto ecologists’ statistical framework
Computer Vision Errors
Biases & Noisein Specific Output
User studies with our visualizations
Duplicated Individuals
Field of View
Develop measurement methods