Passenger Counting / People Counting Applications and Devices
Counting Passenger Vehicles from Satellite...
Transcript of Counting Passenger Vehicles from Satellite...
![Page 1: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/1.jpg)
Counting Passenger Vehicles from Satellite Imagery“ Not everything that can be counted counts, and not everything that counts can be counted”
NVIDIA GPU Technology Conference02 Nov 2017
MAHMOUD LABABIDISENIOR DATA SCIENTIST
KEVIN GREENMACHINE LEARNING AND COMPUTER VISION SCIENTIST
![Page 2: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/2.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 2
“There Are Three Kinds of People -Those Who Can
Count and Those Who Can't”
![Page 3: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/3.jpg)
310/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Agenda Introduction to Satellite Imagery
Why Machine Learning for Satellite Imagery?
Car Counting from Space is Hard▫ Problems▫ Solutions
Algorithm Overview▫ Classification▫ Segmentation▫ Object Detection
Production
Future Work
Car Count = 14 Cars
![Page 4: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/4.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 4
DigitalGlobe’s Satellite Constellation
![Page 5: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/5.jpg)
510/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Why Machine Learning for Satellite Imagery?
Extracting information from imagery in an automated way allows for timely, low-manpower macroscopic as well as needle-in-haystack exploitations (e.g., Intelligence, Business Analytics, Disaster Relief)
Some challenges exist in viewing satellite imagery:▫ Environmental (lighting) and atmospheric (clouds) distortion▫ Pixel count for small objects (cars) in 30 cm resolution imagery
is small (~4-5 pixel width)
Automation exists for basic techniques(e.g., vegetation and land use classification), however, for complex tasks such as object detection, these techniques fail
The capability of transforming overhead imagery content into countable objects will satisfy the analytic needs of commercial and military customers
![Page 6: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/6.jpg)
610/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Car Counting from Space is HardProblems
▫ Counting thousands of cars from 30 cm (1 foot) resolution Satellite imagery is visually difficult (i.e., cars are roughly 5 pixels width), tedious, and laborious
▫ Various brightness levels of cars, some cars are dark, blending into other dark surfaces
▫ Classical computer vision techniques such as edge detection fail in this arena
Our Solutions▫ Object Classification▫ Segmentation + Morphological operations▫ Object Bounding Box Detection
![Page 7: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/7.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 7
Input Imagery
• WorldView 3
Classifier
• LeNet
Morphology
• Binary Threshold
• Convex Hull
• Opening
Labeled Cars and Count
• Estimated Car Count
Algorithm Overview
Segmentation
• Fully Convolutional Network (FCN)
Object Detector
• Single Shot Detector (SSD)
Non-maximum Suppresion
• Removes overlapping detection boxes
Non-maximum Suppresion
• Removes overlapping detection boxes
![Page 8: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/8.jpg)
810/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Algorithm #1: The LeNet Classifier
Yann LeCun started the Convolutional technique to perform classification, particularly for Optical Character Recognition. Their 1998 paper, Gradient-Based Learning Applied to Document Recognition has over 9,600 citations.
![Page 9: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/9.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 9
![Page 10: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/10.jpg)
1010/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
The LeNet Classifier: Sliding Windows and NMS
Trained Classifiers slide a window over an image and placing a bounding box (BBOX) on the portions of the image that contain the object of interest
Non-maximum suppression (NMS) is an algorithm that eliminates overlapping detection boxes that are produced by sliding a window over an image
Sliding Window
Non-maximum Suppression
![Page 11: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/11.jpg)
1110/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
The LeNet Classifier Results
Several issues arise when using an image classifier to detect cars
▫ Determining how much percent overlap to allow before suppressing overlapping detection boxes
▫ More prone to false hits (see figure B)
▫ Challenges in determining car count if BBOX is too large
A)
B)
![Page 12: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/12.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 12
Algorithm #2 Segmentation: Fully Convolution Network
Based on the VGG Neural Network
![Page 13: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/13.jpg)
1310/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Segmentation Results
Here are the issues that surface when using a segmentation algorithm to detect cars
▫ The segmentation outputs exceptional results through pixel masks containing cars but lacks individual BBOXs
▫ Fortunately, segmentation has few false positives to Classification (and Object Detection in some cases)
▫ Although segmentation is powerful, in order to extract individual cars, additional post-processing is needed (e.g., morphology)
B)
A)
![Page 14: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/14.jpg)
1410/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Morphological Operations used to Extract Cars from Segments
![Page 15: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/15.jpg)
1510/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Morphological Operations (Opening/Closing/Convex)
Morphological operations modify the shape of an image in diverse ways:
Erosion – Erodes the boundary of the image
Dilation – Expands the boundary
Opening - Erosion followed by dilation (noise removal)
Closing – Dilation followed by erosion (hole filler)
Convex - Shape formed by a rubber band stretched around foreground image
Image Erosion
Opening
Closing
Convex Hull
Image Dilation
![Page 16: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/16.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 16
Car (RGB) Car (Binary Threshold)
Does this still look like a Car?
![Page 17: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/17.jpg)
1710/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Morphology to Extract Cars
Thresholding followed by morphological operations may not always yield one car blob:
Figure A illustrates two car components after binary thresholding
Figure B does show one car component per car blob after applying an ‘opening’ operation
However, is the resultant car blob really a car?
One final step is validate the car geometry
A)
B)
![Page 18: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/18.jpg)
1810/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Spatial False Alarm Mitigators(FAMs) Reduce False Detections
Spatial FAMs using oriented bounding boxes were used to eliminate blobs that didn’t meet the average dimensions criteria for a car:
Area – Eliminate too big or too small
Length/Width Ratio – Eliminate long skinny boxes
Length and Width – Eliminate if pixel size is one
Area
![Page 19: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/19.jpg)
1910/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Algorithm #2: Segmentation
Conclusion:Segmentation (localization) shows promising results when combined with morphological operations (refinement), enabling us to quickly calculate accurate car counts in satellite imagery
Car Count = 14 Cars
![Page 20: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/20.jpg)
2010/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Algorithm #3: Single Shot Detector (SSD)
![Page 21: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/21.jpg)
2110/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
The SSD Object Detector Results
The Single Shot Detector does a more direct execution at detecting and drawing BBOXs around cars
▫ Locates individual cars more accurately in densely packed parking lots
▫ Also less prone to false hits (see figure B)
▫ NMS issues may arise if overlap area isn’t calibrated
A)
B)
![Page 22: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/22.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 22
Competitive Car-Counting Bake-off
Other Commercial Vendor = 2,205
![Page 23: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/23.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 23
Training (wheels) to Production
NVIDIA GPU Training:NVIDIA GTX 980NVIDIA GTX 1080NVIDIA TitanXNVIDIA M40NVIDIA P100NVIDIA M1000M - Mobile
Machine Learning Frameworks:TensorFlowKerasCaffe
GBDX is the platform that uses Amazon EC2 to deploy docker images of the code and model.
Training Speed: 100 Batches of 4 300x300 images take 20 minutes to train
Inference Speed: 20 minutes on strip shown above approximately 13kx13k
![Page 24: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/24.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 24
AnswerFactory – SSD Workflow1) Define AOI & Select Detect Model (Cars) 2) Select Date Ranges & Auto Update
Historical (15 years) Run on all new
images in the future
AnswerFactory
VIP &
Visitor
Parking
Resident
Parking
Employee
Parking
4) Analyze Individual Parking Lots Over-time3) Run Model & Get Results
![Page 25: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/25.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 25
Provide analysts “Tips” on changing activity levels for enhanced garrison monitoring
0
50
100
150
200
250
300
350
16APR2015 29AUG2015 2SEP2015
Military Vehicle Counting
Object Count
Actual Count
16APR2015
29AUG2015
2SEP2015
Significant ChangesDetected
![Page 26: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/26.jpg)
2610/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.
Future Work
Explore the use of different multi-spectral band combinations for improved car count
Explore whether different activations might better support detecting dark cars (e.g., Leaky Relu)
Go beyond temporal volume anomalies to include spatial anomalous behaviour
Upcoming Xview Challenge, which is an ImageNet-like challenge competition for Satellite imagery
DigitalGlobe colleague, Tood Bacastow, discussed earlier today in his talk entitled “SpaceNet: Accelerating Automated Mapping with Deep Learning and Labeled Satellite Imagery”
![Page 27: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/27.jpg)
10/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved. 27
Thank you!Questions?
![Page 28: Counting Passenger Vehicles from Satellite Imageryon-demand.gputechconf.com/gtcdc/2017/presentation/...Counting Passenger Vehicles from Satellite Imagery “ Not everything that can](https://reader033.fdocuments.us/reader033/viewer/2022041909/5e6696e688eede3eaa327548/html5/thumbnails/28.jpg)
2810/31/2017 Copyright © DigitalGlobe | Radiant. All rights reserved.