Cooperative Unmanned Vehicles for Vision-based ... · HOG1-Based Human Classification...
Transcript of Cooperative Unmanned Vehicles for Vision-based ... · HOG1-Based Human Classification...
Cooperative Unmanned Vehicles for Vision-based Detection and Real-World
Localization of Human Crowds Sponsor:
Air Force Office of Scientific ResearchDDDAS Program (Dr. Darema)
Sara Minaeian, Dr. Jian Liu, Dr. Young-Jun Son
Systems and Industrial Engineering,The University of Arizona
November 3, 2015
Computer Integrated Manufacturing & Simulation Lab
Agenda
• Scope of the Crowd Control Project
• Crowd Detection by UAV and UGV
• Real-world Localization
• System Implementation
• Experiment and Results
• Summary and Ongoing Works
1
Computer Integrated Manufacturing & Simulation Lab
Problem Motivation• Motivation: TUS-1 BP Project (23-mile long border area of Sasabe, AZ)
• Problem: highly complex, uncertain, dynamically changing environment
• Main goal: create scalable, robust, multi-scale, and effective surveillance& crowd control strategies using collaborative UVs
• Proposed approach: a comprehensive planning and control frameworkbased on dynamic-data-driven, adaptive multi-scale simulation (DDDAMS)
21.Reuters, FY2000 ~ FY2013 : http://graphics.thomsonreuters.com/14/immigration/index.html
1
1
Computer Integrated Manufacturing & Simulation Lab
3
DDDAMS-based Framework
1. Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. J., & Liu, J. (2013, December). Agent-based hardware-in-the-loop simulation for UAV/UGV surveillance and crowd control system. In Proceedings of the 2013 Winter Simulation Conference (pp. 1455-1466). IEEE Press.
1
Computer Integrated Manufacturing & Simulation Lab
• Objective of Detection Module: vision-based detection ofthe moving targets (crowd) dynamically at each time stamp t
• Objective of Localization: finding the real-world location ofthe detected targets and send them to the Tracking module forpredicting their locations at
4
Crowd Detection Module
Crowd Tracking Module
Motion Planning ModuleDetected crowd locations
at time tPredicted crowd locations
at time .
Scope of the Project
t t
t t
x(t ), y(t )
Computer Integrated Manufacturing & Simulation Lab
Agenda
• Scope of the Crowd Control Project
• Crowd Detection by UAV and UGV
• Real-world Localization
• System Implementation
• Experiment and Results
• Summary and Ongoing Works
5
Computer Integrated Manufacturing & Simulation Lab
UAV-UGV Cooperation in Detection• Cooperation in detection & localization of the moving crowd
• Unmanned Aerial Vehicle (UAV)– Further distance– Lower resolution– Top-down perspective of moving objects– More appropriate for Crowd detection– Optical-flow-based Motion Detection
6
• Unmanned Ground Vehicle (UGV)– Closer distance– Higher resolution– Upright perspective of human– More appropriate for Individual detection– HOG-based Human Classification
Computer Integrated Manufacturing & Simulation Lab
1. Extract featured keypoints from frames t using GFTT1
Crowd Detection Module UAV (1)
71. J. Shi and C. Tomasi, “Good features to track,” in Computer Vision and Pattern Recognition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, 1994, pp. 593–600.
Iui
2i Iui
IviiIui
Ivii Ivi
2i
• Autocorrelation matrix of the second derivative images• Good features: 2 eigenvalues greater than a minimum threshold
Computer Integrated Manufacturing & Simulation Lab
2. Track (match) keypoints over frames t+1 and t+2 using Optical FlowMatched keypoints over frame t+1 Matched keypoints over frame t+2
Crowd Detection Module UAV (2)
3. Perspective warp of frames t+1 and t+2 onto frame t using HomographyWarped frame t+1 over t Warped frame t+2 over t
8
Computer Integrated Manufacturing & Simulation Lab
I (T)t1 Ht It1
• (3.1) Apply RANSAC on each pairs of frames for filtering out the outliers (keypoints belonging to the foreground)
• (3.2) Estimate the Homography*
between frames t and t+1; & between frames t and t+2 :
• (3.3) Warp frames t+1 and t+2 onto frame t based on perspective transformation using H matrix:
H
h11 h12 h13
h21 h22 h23
h31 h32 h33
Crowd Detection Module UAV (3)2 Frames: Compensate the background motion and reduce the registration error;
By referencing the same Homography
9
It1 Ht It1 It2 Ht It2
I (T)t2 Ht It2
* In the field of computer vision, any two images ofthe same planar surface are related by a Homography:
Computer Integrated Manufacturing & Simulation Lab
• (5.1) Make the silhouette– By applying threshold
• (5.2) Smoothen the image– By applying Gaussian filter
• (5.3) Dilate* the image– To fill small holes in the motion area
• (5.4) Erode* the image– To remove small and separated noise
4. Take absolute differences between transformed frames t+1 and t+2
Dilation
-
Erosion
* Morphological transformation applying a Kernel
Crowd Detection Module UAV (4)
10
Computer Integrated Manufacturing & Simulation Lab
5. Segment the moving foreground and assign target boundaries: Detection
Crowd Detection Module UAV (5)
11
Computer Integrated Manufacturing & Simulation Lab
Individual Detection Module UGV
: OpenCV classifier
12
HOG1-Based Human Classification : 3x3 derivative mask[-1, 0 , 1]
: Weighted voting over cells6x6 pixel cells
: Grouping cells into blocks3x3 cell blocks
: L2-norm
Gradient
computation
Orientation
binning
HOG over des. blocks
Block normalization
Classify the
target
1. Dalal, N., & Triggs, B. (2005, June). Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on (Vol. 1, pp. 886-893). IEEE.
Blocks
Cells
Computer Integrated Manufacturing & Simulation Lab
Agenda
• Scope of the Crowd Control Project
• Crowd Detection by UAV and UGV
• Real-World Localization
• System Implementation
• Experiment and Results
• Summary and Ongoing Works
13
Computer Integrated Manufacturing & Simulation Lab
Challenges for Target Localization via UAV
No clue (color, template, size) of the target to detect.
Moving camera and changing background.
Unknown transformation between Image plane and
Earth plane.
Vertical positioning error of the UAV, as well as
Lateral positioning error.
Lateral positioning error of the UGV only.
Computer Integrated Manufacturing & Simulation Lab
• Perspective transformation between two planes 1
• Preserves collinearity
151. Criminisi, A., Reid, I., & Zisserman, A. (1999). A plane measuring device. Image and Vision Computing, 17(8), 625-634.
A’ B’
C’D’
D
A B
C
O: center of projection
u ax by rpx qy 1
v cx dy spx qy 1
X=MU(x, y)(u,v) : Image position
: World locationM :Transform matrix
(8 parameters)
Landmark-based Localization
Computer Integrated Manufacturing & Simulation Lab
Localization Framework (IEEE SMC) 1
16
Landmarks with: (x,y): known GPS location, (u,v): detected image location.
UAV
UGVs as colored landmarks
UAV’s detection range
• At least 4 coplanar, non-collinear landmarks• UGVs with their real-world location (GPS) and image position (colored) known
• UAV is flying high enough,• UGVs and crowd are close enough,• Disregard any internal depth
differences between the UGVsand crowd on earth’s plane.
UAV’s detection range
UGV’s detection range
Crowd’s individuals
UGV
UAV
Weak Perspective approximation:
1. Minaeian, S, Liu, J., & Son, Y. Vision-based Target Detection and Localization via a Team of Cooperative UAV and UGVs, IEEE Transactions on Systems, Man, and Cybernetics (Accepted).
Computer Integrated Manufacturing & Simulation Lab
Localization Algorithm UAV• Landmark ( ) with image position ( ) at time t and
real-world geographic location ( ) at time t• Using homogeneous coordinates• Transformation matrix between two planes:
where: and
• System of linear equations: where
17
i 1,...,ni ui (t ), vi (t )
xi (t ), yi (t )
Ui (t )
Vi (t )
Wi (t )
M
xi (t )
yi (t )
1
M
ui (t ) Ui (t ) Wi (t )
vi (t ) Vi (t ) Wi (t )
M
a b rc d sp q 1
Am k m a, b, r, c, d, s, p, q T
Computer Integrated Manufacturing & Simulation Lab
Solving the System of Equations• Three alternatives:
– If Under-determined
– If Exact solution:
– If Linear Least Squares:
Inverse method Not very stable numerically & expensive Gaussian elimination/ Back substitution Less expensive
• For any detected target with image position ( ) andunknown GIS-based location ( ) at time t
18
n 4; n m A-1k
n 4; n m (AT A)1AT k
u (t ), v (t )
x (t ), y (t )
n 4; n
Computer Integrated Manufacturing & Simulation Lab
Landmark Assignment Problem• UGVs (as landmarks) are also moving• How to differentiate these landmarks from the moving targets?
Using an Assignment Model:
Mathematical Model(m>n)
Minimizing the Total Errorof Landmark Assignment
Euclidean distance between ith color-detected and jth motion-detected objects
Decision variable to relate corresponding motion-detected objects and landmarks
i-j pair assignment constraints
min Dijzijj1
mi1
ns.t. zijj1
m 1 i 1,...,n
ziji1
n 1 j 1,...,m
zij 0,1 i, j
19
Computer Integrated Manufacturing & Simulation Lab
• UGVs also need to transform image coordinates to real-world positions
• Compute camera’s calibrated focal length:
• Estimate target j ’s distance from the UGV:
• UGV’s camera pose:
• Target j ’s real-world pose (location + orientation):
20
Longj , Lat j LongC Dj sin( j ), LatC Dj cos( j )
f D hH
Dj f Hhj
LongC , LatC ,C
j C tan1(devj f ) j
Localization Heuristic UGV
C jDj cos( j )
Dj sin( j )
Lat
Long
(decimal degree)
(decimal degree)
DjLongC , LatC
Target j
UGV’s Camera
Computer Integrated Manufacturing & Simulation Lab
Agenda
• Scope of the Crowd Control Project
• Crowd Detection by UAV and UGV
• Real-world Localization
• System Implementation
• Experiment and Results
• Summary and Ongoing Works
21
Computer Integrated Manufacturing & Simulation Lab
Real Testbed for Detection
Carl Zeiss Tessar HD1080pHD (16:9): 1280x720p @ 30 fpsFOV: 90
22
GoPro HERO 3+ Tarot GimbalHD (16:9): 1280x720p @ 120 ~ 25 fpsFOV(x): 64.4 ; FOV(y): 37.2
DR(x)
DR(y)
FOV (x)
FOV(y)
h
Onboard Computer: ODROID U31.7 GHz quad-core ARM-Cortex-A9 Linux-based operating system
Computer Integrated Manufacturing & Simulation Lab
Simulated Testbed for Localization
23
• No adequate number of real UGVs to run the localization algorithm• Higher costs of conducting experiments on the border area of Tucson• UGVs and crowd need to move randomly as independent agents in a simulated
border area (using GIS information)• Platform: Repast Simphony (Open source, Java, NASA World Wind SDK)
Computer Integrated Manufacturing & Simulation Lab
Agent-based SimulationRepast Simphony with 3D GIS
UAV(APM:Copter / Arducopter)
UGV(APM:Rover / Ardurover)
Sensory Data (e.g. GPS)
Control Commands (MAVLink Messages)
Hardware Interface
Hardware-in-the-Loop Testbed
HOG-BasedHuman Classification
Optical-Flow-Based Motion Detection
24
Computer Integrated Manufacturing & Simulation Lab
Agenda
• Scope of the Crowd Control Project
• Crowd Detection by UAV and UGV
• Real-world Localization
• System Implementation
• Experiment and Results
• Summary and Ongoing Works
25
Computer Integrated Manufacturing & Simulation Lab
Agent-based Simulation for Localization• Random movements of the crowd in a N-E path in border area of Tucson, AZ• Changing parameters:
– Flight altitude– Number of landmarks– Landmark assignment method
26
Motion-Detected Crowd
Color-Detected
UGVsAgent-based Simulation
Movement speed :1 ~ 3 m/s
Detection & LocalizationAlgorithm
Computer Integrated Manufacturing & Simulation Lab
Experiments: Localization (Altitude)
27
31.3318
31.3319
31.3320
31.3321
31.3322
31.3323
31.3324
-110.9046 -110.9045 -110.9044 -110.9043 -110.9042 -110.9041 -110.9040 -110.9039
Lat
itude
(dec
imal
deg
ree)
Longitude (decimal degree)
Simulated Target Detected Target (at 50 m) Detected Target (at 200 m)
0
2
4
6
8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
Euc
lidea
n E
rror
(M
eter
s)
Simulation Time Stamp (tic)
Error for 50 m Altitude Error for 200 m Altitude
Average Euclidean Error: 3.5 m
Average Euclidean Error: 1.3 m 63% improvements
Computer Integrated Manufacturing & Simulation Lab
Experiments: Localization (Landmarks #)
28
31.3318
31.3319
31.3320
31.3321
31.3322
31.3323
31.3324
-110.9046 -110.9045 -110.9044 -110.9043 -110.9042 -110.9041 -110.9040 -110.9039
Lat
itude
(dec
imal
deg
ree)
Longitude (decimal degree)
Simulated Target Detected Target (4 Landmarks) Detected Target (6 Landmarks)
0.000
0.002
0.004
0.006
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Err
or (K
ilom
eter
s)
Simulation Time Stamp (tic)
Error for 4 Landmarks Error for 6 Landmarks
Average Euclidean Error: 2.5 m
Average Euclidean Error: 2.2 m
Euc
lidea
n E
rror
(M
eter
s)
6
4
2
0
12% improvements
Computer Integrated Manufacturing & Simulation Lab
Experiments: Localization (Assignment)
29
32% improvements:motion-based landmark assignment
versuscolor-based landmark assignment
0.000
0.001
0.002
0.003
0.004
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Err
or (K
ilom
eter
s)
Simulation Time Stamp (tic)
Error for 6 Landmarks; Color-based Error for 6 Landmarks; Motion-based
Error for 4 Landmarks; Color-based Error for 4 Landmarks; Motion-based
Euc
lidea
n E
rror
(M
eter
s)
4
3
2
1
0
2.24
2.49
3.52
3.48
0.0 1.0 2.0 3.0 4.0 5.0
Average Euclidean Error (Meters)
Colored-based assignment: 6 Landmarks
Colored-based assignment: 4 Landmarks
Motion-based assignment: 6 Landmarks
Motion-based assignment: 4 Landmarks
Computer Integrated Manufacturing & Simulation Lab
30
Stationary Aerial Non-Stationary Aerial
Camera Pose Cliff et al., 2015 Farmani et al., 2014Redding et al., 2006
Fixed Landmarks Martinez et al., 2011Wang et al., 2014
Dementhon & Davis: POSIT
Moving Landmarks ---
Literature on Target’s Geo-Localization
Contribution of the Work (IEEE SMC) 1
1. Minaeian, S, Liu, J., & Son, Y. Vision-based Target Detection and Localization via a Team of Cooperative UAV and UGVs, IEEE Transactions on Systems, Man, and Cybernetics (Accepted).
Computer Integrated Manufacturing & Simulation Lab
Agenda
• Scope of the Crowd Control Project
• Crowd Detection by UAV and UGV
• Real-world Localization
• System Implementation
• Experiment and Results
• Summary and Ongoing Works
31
Computer Integrated Manufacturing & Simulation Lab
Summary and Ongoing WorksCrowd Detection and Localization using Team of UVs
o Collaborative crowd detection: Motion detection (UAV) Human classification (UGV)
o Collaborative localization: Landmark-based localization (UAV) Heuristic localization (UGV)
o Testbed and experiments: Hardware-in-the-Loop Agent-based simulation Sensitivity Analysis
o Ongoing Work: Human-in-the-loop Hardware-in-the-loop Simulation
32
Computer Integrated Manufacturing & Simulation Lab
33
Sara Minaeian: [email protected]. Jian Liu: [email protected]. Young-Jun Son: [email protected]