Evaluation Methods for Curvilinear Featuresafeworks.pbworks.com/f/Evaluation+methods.pdf · 1...
32
1 Approved for public release NGA 09-519 Evaluation Methods for Curvilinear Features Peter Doucette 2 , Ann Martin 1 , Chris Kavanagh 2 , Stephen Barton 2 , Tim McIntyre 2 , Jacek Grodecki 3 , Josh Nolting 3 , Seth Malitz 3 1 National Geospatial-Intelligence Agency (NGA) 2 Contractor for NGA 3 GeoEye Approved for public release NGA 09-519
Transcript of Evaluation Methods for Curvilinear Featuresafeworks.pbworks.com/f/Evaluation+methods.pdf · 1...
1
Approved for public release NGA 09-519
Evaluation Methods for Curvilinear Features
Peter Doucette2, Ann Martin1, Chris Kavanagh2, Stephen Barton2, Tim McIntyre2, Jacek Grodecki3, Josh Nolting3, Seth Malitz3
1National Geospatial-Intelligence Agency (NGA)2Contractor for NGA3GeoEye
Approved for public release NGA 09-519
2
Approved for public release NGA 09-519
Objectives
• Automating the extracting/delineating of landcover features from imagery into GIS-vector layers, i.e., AFE (Automated Feature Extraction)
• Determining the Effect on Productivity (EOP) for an AFE tool
3
Approved for public release NGA 09-519
Algorithms evaluated
• GeoEye RoadTracker® • Commerical plug-in for Feature Analyst
(OverWatch)• Includes extraction strategies for:
– interactive– full automation– updating existing vectors– “smart” editing
4
Approved for public release NGA 09-519
Interactive Road Tracker (IRT)Source image: IKONOS © GeoEye
VideoDemonstration
5
Approved for public release NGA 09-519
Automated Road Tracker (ART)
Source image: IKONOS © GeoEye
6
Approved for public release NGA 09-519
Vector updateTIGERDCGIS
Image source: DCGIS 0.3m RGB aerialVector source: DCGIS (open source)
TIGER (Census Bureau)
7
Approved for public release NGA 09-519
Smart editingSource image: IKONOS © GeoEye
8
Approved for public release NGA 09-519
Vector update and editing
Source image: IKONOS © GeoEye
Video demonstration
9
Approved for public release NGA 09-519
Types of evaluation
• Accuracy: compare spatial accuracy of extraction between human and computer.
• Effect On Productivity (EOP): timed comparison between conventional and computer assisted extraction.
10
Approved for public release NGA 09-519
Timed comparison methods
• Training and practice with automation tools prior to testing.
• Limit bias of learning effects by changing order of extraction modes.
• User time includes extraction and editing, but not off-line processing.
11
Approved for public release NGA 09-519
Accuracy evaluation Image: USGS 0.3m RGB aerialVector: DCGIS (open source)
12
Approved for public release NGA 09-519
Different operating conditions
Operating conditionsfor evaluation
Source: Image USGS 0.3m RGB aerialVector: DCGIS (open source)
13
Approved for public release NGA 09-519
Reference layer Source: Image USGS 0.3m RGB aerialVector: DCGIS (open source)
14
Approved for public release NGA 09-519
Reference buffer (5m)ref
buffer
Source: Image USGS 0.3m RGB aerialVector: DCGIS (open source)
15
Approved for public release NGA 09-519
FP
Output categories
buffer
TP
FN
Source: Image USGS 0.3m RGB aerial
16
Approved for public release NGA 09-519
“Leaf off” results (5m buf)
TP
FN
FP
Source: Image USGS 0.3m RGB aerial
17
Approved for public release NGA 09-519
Leaf-off portion of imageEntire image
Scoringmetrics
18
Approved for public release NGA 09-519
Limitations of buffer approach Reference Data Vectors for comparison
overly optimisticUNCLASSIFIED
UNCLASSIFIED
19
Approved for public release NGA 09-519
Hypotheses
• H1. Interactive extraction is more beneficial to EOP than full automation (i.e., on-the-fly editing versus post-process editing)
• H2. Correctness is more beneficial to EOP than completeness
• H3. “Smart editing” is more beneficial to EOP than conventional editing
20
Approved for public release NGA 09-519
What is truth? (source: RGB aerial, GSD = 0.3m, USGS )Operator 1 Operator 2
21
Approved for public release NGA 09-519
Timed comparisons
22
Approved for public release NGA 09-519
0
1
2
3
4
5
6
0 1 2 3 4 5 6
Automation-based (hours)
Con
vent
iona
l (ho
urs)
Op1 IRTOp1 ARTOp2 IRTOp2 ARTOp3 IRTOp3 ART
User timings test 1
Conventionalis better
Automationis better
23
Approved for public release NGA 09-519
Test scene 2Extraction Time
050
100150200250300350
Op1 Op2
Manual IRT ART edit
Consistency of Extraction for Scene 2
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10
Reference Buffer (m)
Perc
enta
ge (%
)
Op1 Man
Op2 Man
Op1 IRT
Op2 IRT
Op1 ART edit
Op2 ART edit
ART no edit (correctness)
ART no edit (completeness)
Presenter
Presentation Notes
In clean Panchromatic images, there is very little variance between operators and methods. High resolution panchromatic imagery is easy for humans to determine which features to extract, resulting in each human extraction to have nearly 100% correlation. What appears to be a single yellow line is a result of the assisted method extracting the same features nearly perfect regardless of which operator, or in other words, which seed points were chosen for feature extraction. The tool was also very good at extracting features based on seed points, as shown by the reduction in time vs manual. The automated algorithm performed fairly well. If the human threshold of precision is 2m, then the automated algorithm would have completed ~65% of the features at nearly 60% accuracy overall, relieving the human of a lot of time required to extract from a blank slate and instead focus on editing false positives and completing the remaining 35% of the extraction necessary. This is evident in the amount of time the edited methods required vs either method that required the operator to start from scratch.
24
Approved for public release NGA 09-519
Consistency of Extraction for Scene 3
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10
Buffer (m)
Perc
enta
ge (%
)
Op1 Man
Op2 Man
Op1 IRT
Op2 IRT
Op1 ART edit
Op2 ART edit
ART no edit(correctness)ART no edit(completeness)
Test scene 3 Extraction Time
0
50
100
150
200
Op1 Op2
Man IRT Edit
25
Approved for public release NGA 09-519
User timings test 2
0
50
100
150
0 50 100 150
Smart Editing (minutes)
Con
vent
iona
l Edi
ting
(min
utes
)
Scene 1
Scene 2
26
Approved for public release NGA 09-519
Vector update performance
0.0
10.0
20 .0
30 .0
40 .0
50 .0
60 .0
70 .0
80 .0
90 .0
100 .0
0 1 2 3 4 5 6 7 8 9 10
R an g e fro m C en terlin e R e fe ren ce (m )
Con
sist
ency
(%)
no perturbation
5m (x,y s hift), 0m (x,y drift)
5m , 2.5m
10m , 0m
10m , 5m
15m , 0m
15m , 7.5m
27
Approved for public release NGA 09-519
Editing: post-process vs. on-the-fly(source: QB PAN, GSD = 1m, © DigitalGlobe )Algorithm Edited
28
Approved for public release NGA 09-519
Findings for automation methods• Even if automation methods take more time
than conventional methods, its output can be more precise and consistent.
• Batch automation is optimal for “clean” suburban streets typical of North American environments, but suboptimal for less developed urban environments.
• Batch automation may be more effective at favoring correctness over completeness.
29
Approved for public release NGA 09-519
• truth is relative
• although quantitative methods are inherently empirical, they can provide meaningful insights for developer and user
• ultimately the proof is in the pudding--monitor usage rates of automation tools
Findings for evaluation methods
30
Approved for public release NGA 09-519
Future directions: sketch and snapSource: Image USGS 0.3m RGB aerial
31
Approved for public release NGA 09-519
Future directions: sketch and snapSource: Image USGS 0.3m RGB aerial
32
Approved for public release NGA 09-519
www.nga.mil
