A Hierarchical and Scalable Situation Awareness System … · A Hierarchical and Scalable Situation...

A Hierarchical and Scalable Situation Awareness Systemfor 3-D Border Surveillance

Sponsor: Air Force Office of Scientific Research

FA9550-12-1-0238 (DDDAS); 15RT1016 (New)Program Manager: Dr. Frederica Darema

PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2

Students: S. Minaeian1, Y. Yuan1, S. Lee1, and J. Han1

1Systems and Industrial Engineering, University of Arizona2Computer Science, George Mason University

PI Contact: [email protected]; 1-520-626-9530

AFOSR DDDAS PI Meeting Jan. 2016

Computer Integrated Manufacturing & Simulation LabDepartment of Systems and Industrial Engineering, The University of Arizona, Tucson

Agenda

• Previous project• New project

– New challenges– How DDDAS is addressed


Overview of Previous Project

Goal: Develop a simulation-based planning and control system forsurveillance and crowd control via collaborative UAVs/UGVs

Motivation: TUS 1- Project (23-mile long area of the border in Sasabe, AZ)

Problem: A highly complex, uncertain, dynamically changing border environment


DDDAMS-based Planning and Control Framework

Khaleghi, A. M., Xu, D., Wang, Z., Li, M., Lobos, A., Liu, J., & Son, Y. (2013). A DDDAMS-based Planning and Control Framework for Surveillance and Crowd Control via UAVs and UGVs. Expert Systems with Applications, 40, 7168-7183.


Multi-resolution Data

Challenge: Aggregate multi-resolution data

Approach: UAVs’ global perception and UGVs’ detailed perception


Framework of Developed Methodology


DR(x)

DR(y)

FOV (x)FOV(y)

h

Detection Module: Simulated

DRminG 2 hmin

G tan FOV / 2 DR G 2 hmaxG tan FOV / 2 DRmax

G DR( A) 2h( A) tan FOV / 2

GoPro HERO 3- Tarot Gimbal StabilizerHD (16:9): 1280x720p @ 120 ~ 25 fpsFOV(x): 64.4 ; FOV(y): 37.2

EDD : UGV’s Effective Detection Depth : Detection range for UGV: Detection range for UAV: Field of view: Distance: Altitude

DR G

DR A

FOV

h( A)h(G )

ODROID USB-CAM 720PHD (16:9): 1280x720p @ 30 fpsFOV: 72


Detection Module (UAV & UGV): Actual


Crowd Tracking Module


Case Study

Bayesian estimation: 75% Less computation timeComparable/Better prediction performance


Control Strategies associated with Motion Planning

• Given selected destination of UAV/UGV, find the path that optimizes a certain combination of criteria

• Weighted average of the multiple objectives

(c) minimize the weighted average of (a) and (b)

(a) minimize travel distance (b) minimize elevation penalty (fuel consumption)


System Implementation• Agent-based HIL simulation• UAVs and UGVs• Social force model and GIS


Agent-based Hardware-in-the-loop SimulationAgent-based Simulation

Repast Simphonywith 3D GIS

Wi-Fi / XBee PRO 900HP; APM one

Assembled UAV (APM:Copter / Arducopter)

Assembled UGV (APM:Rover / Ardurover)

Sensory Data (e.g. GPS)

Control Commands (MAVLink Messages)

Hardware Interface:MAVproxy

Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA.


Assembled UAV (Arducopter), AR.Drone, X8+ UAV, and UGV

Navigate GPS waypoints autonomously using APM autopilot set (Arduino-based Autopilot- APM 2.5)

Motion Processing Unit (MPU-6000)3-Axis Gyro3-Axis Accelerometer

Microcontroller(ATMEGA2560)Low-power Atmel 8-bit AVR RISC-based256KB ISP flash memory8KB SRAM4KB EPROMThroughput: 16 MIPS at 16MHz

Barometric pressure sensor (SM5611)

GPS update rate: 5hz (5X per second)Using GPS unit, the UAV has an outdoor navigation accuracy of about +/- 5 meters

Global Positioning System (GPS)


Social Force Model for Crowd MotionDirection/angle and walking speed of humans has been modeled using 2 heuristics based on their visual data (Moussaïd et al., 2011)

• Heuristic 1: Minimize the angle/direction of each individual by minimizing the distance to its destination

• Heuristic 2: Change walking speed of human to avoid collisionsmin d( ) dmax

2 f 2( ) 2dmax f ( )cos( 0 )

v min(v0, dh

)

human field of view: (, ) (for e.g. 90 ,90)maximum range of view: dmax (for e.g. 10 m)human comfortable walking speed: v0 (for e.g. 1.5 m / s)distance to obstacle: dh

relaxation time time requires to adopt new behavior : (for e.g. 1 sec)

Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA.


Publications: Journals and Book Chapters• S. Minaeian, J. Liu, Y. Son, Vision-based Target Detection and Localization via a Team

of Cooperative UAV and UGVs, IEEE Transactions on Systems, Man, and Cybernetics: Systems (Special Issue on Biomedical Robotics and Bio-mechatronics Systems and Application), Accepted, 2015.

• A. Khaleghi, D. Xu, Z. Wang, M. Li, A. Lobos, J. Liu, Y. Son, A DDDAMS-based Planning and Control Framework for Surveillance and Crowd Control via UAVs and UGVs, Expert Systems with Applications, 40, 2013, 7168-7183.

• Yifei Yuan, Zhenrui Wang, Mingyang Li, Young-Jun Son, Jian Liu, DDDAS-based Information-Aggregation for Crowd Dynamics Modeling with UAVs and UGVs, Frontiers in Robotics and AI (Sensor Fusion and Machine Perception Section), 2:8, 2015, 1-10.

• A. Khaleghi, D. Xu, S. Minaeian, M. Li, Y. Yuan, C. Vo, A. Mousavian, J. Lien, J. Liu, and Y. Son, UAV/UGV Surveillance and Crowd Control via Hardware-in-the-loop DDDAMS System, Darema, F., Douglas, C (Eds.), Springer (under review)

• Online Collision Prediction Among 2D Polygonal and Articulated Obstacles, Yanyan Lu, Zhonghua Xi and Jyh-Ming Lien, International Journal of Robotics Research (IJRR), Accepted, 2015.


Publications: Proceedings (1)• Minaeian, S., Yuan, Y., Liu, J., and Son, Y., 2015, “Human-in-the-Loop Agent-based

Simulation for Improved Autonomous Surveillance using Unmanned Vehicles,” Proceedings of 2015 Winter Simulation Conference, Huntington Beach, CA (poster)

• Continuous Visibility Feature, Guilin Lu, Yotam Gingold, and Jyh-Ming Lien, in the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, Boston, MA, USA

• Semantically Guided Location Recognition for Outdoors Scenes, Arsalan Mousavian, Jana Kosecka and Jyh-Ming Lien, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), May 2015, Seattle, WA, USA

• Khaleghi, A., Xu, D., Minaeian, S., Yuan, Y., Liu, J., and Son, Y., 2015, “Analysis of UAV/UGV Control Strategies in a DDDAMS-based Surveillance System,” Proceedings of 2015 IIE Annual Meeting, Nashville, TN.

• Minaeian, S., Liu, J., and Son, Y., 2015, “Crowd Detection and Localization Using a Team of Cooperative UAV/UGVs,” Proceedings of 2015 IIE Annual Meeting, Nashville, TN.

• Khaleghi, A., Xu, D., Minaeian, S., Li, M., Yuan, Y., Liu, J., Son, Y., Vo, C., and Lien, J., 2014, “A DDDAMS-based UAV and UGV Team Formation Approach for Surveillance and Crowd Control,” Proceedings of 2014 Winter Simulation Conference, Savannah, GA.


Publications: Proceedings (2)• Khaleghi, A., Xu, D., Minaeian, S., Li, M., Yuan, Y., Liu, J., and Son, Y., 2014, “A

Comparative Study of Control Architectures in UAV/UGV-based Surveillance System,” Proceedings of 2014 IIE Annual Meeting, Montreal, Canada.

• Wang, Z., Li, M., Khaleghi, A., Xu, D., Lobos, A., Vo, C., Lien, J., Liu, J., and Son, Y., 2013, “DDDAMS-based Crowd Control via UAVs and UGVs,” Procedia Computer Science 18, 2028–2035 (Proceedings of 2013 International Conference on Computational Science, Barcelona, Spain).

• Khaleghi, A., Xu, D., Lobos, A., Minaeian, S., Son, Y., and Liu, J., 2013, “Agent-based Hardware-in-the-Loop Simulation for Modeling UAV/UGV Surveillance and Crowd Control System,” Proceedings of 2013 Winter Simulation Conference, Washington DC.

• Wang, Z., Li, M., Khaleghi, A., Xu, D., Lobos, A., Vo, C., Lien, J., Liu, J., and Son, Y., 2013, “DDDAMS-based Crowd Control via UAVs and UGVs,” Procedia Computer Science 18, 2028–2035 (Proceedings of 2013 International Conference on Computational Science, Barcelona, Spain).

• Vo, C., McKay, S., Garg, N., and Lien, J., 2012, “Following a Group of Targets in Large Environments”, Proceedings of the Fifth International Conference on Motion in Games, Springer.

• Vo, C., and Lien, J., 2012, “Group Following in Monotonic Tracking Regions”, Proceedings of the 22nd Fall Workshop on Computational Geometry, 2012.


New Project• New challenges• How DDDAS is addressed


3-Level Surveillance Framework

High altitude

level (HAL)

Low altitude

level (LAL)

Surface level (SL)


New Major Problems

• 3-D Surveillance System for aerial and ground targets

• Latency in detection, recognition and identification of targets

• Heterogeneous data from complex targets by 3 levels of sensors

• Multi-level information aggregation

• Active or pro-active surveillance strategies

• Realistic scenarios and model validation based on data collection from our research partners (AFRL, Raytheon, University Partners …)


High altitude level (HAL)

Electro-Optical/Infrared

(EO/IR)

Synthetic Aperture Radar

(SAR)

Low altitude level (LAL)

Remote Sensing

Surface level (SL)

Mobile Sensors

Fixed Sensors

3-Level Measurement System in Border Surveillance

Surveillance Camera

3-Levels Types Sensors Measurement Data

SAR Image

EO/IR Image

Lidar Image

Thermal Images

Magnetic Data

Spectral Image


A generic modeling framework of sensors for the surveillance application

Adjust the Detection range;Set the threshold

Receive signals / messages from the controller

Change location parameters of sensors in order for chasing foe targets

Target appears in the system

Surveillance Behavior Sub-Model

Availability Sub-Model

Signal Processing Sub-Model

Agent Model of Sensors (Generic)


Enhanced DDDAMS-based FrameworkPrevious DDDAMS-based Framework


1.0 - Target DRI Detection, Recognition, and Identification of hostile targets (i.e. traffickers)

and civilian targets.

* Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense.

Discovery the presence of a person,object, or phenomenon*

• Sensing technologies: thermal technologies, radar, etc.• Motion detection, optical flow, etc.


1.0 - Target DRI


Determination of the nature of a detected person,object or phenomenon, and its class or type *

• Image Feature Extraction• Functional Discriminant Analysis for Time series sensor data• Information-aggregation method for Multiple Sensor data• Multivariate Classification Method

Detection, Recognition, and Identification of hostile targets (i.e. traffickers)and civilian targets.


1.0 - Target DRI


Discrimination between recognizable objects asbeing friendly or enemy *

• Gaussian Mixture Model for Objective Identification• BDI (Belief–Desire–Intention) framework

Detection, Recognition, and Identification of hostile targets (i.e. traffickers)and civilian targets.


Sensor Type Vehicle Armed People Animals

Image

Thermal Sensor

Magnetic Sensor

Underground, Radar, etc

… … …

Information Aggregated Classifier

Training

New Observation

Predicted Class

Real Class

Updating

Sample Image DataChallenge:Feature Extraction

Potential Method:Discriminant Analysis Fe

atur

e 2

Feature 1

VV VV VV

VA

A AAA

A AAAP APAP APAPAP

Challenge: Classification underDDDAS framework

Potential Methods:SVM, KNN, etc.

Emerging Challenges - Target DRI


Bratman, 1987Rao and Georgeff, 1998Zhao and Son, 2008

Extended Belief-Desire-Intention Framework

Lee, S., Y.-J. Son, and J. Jin (2010), Integrated human decision making and planning model under extended belief-desire-intention framework, ACM Transactions on Modeling and Computer Simulation, 20(4), 23(1)~23(24).


Time

Location

Route Choice En-Route Planning

Fastest Way

Rugged Road

Selection at Departure

Hiding/Stopping

Sudden Direction Change

Decision 1 Decision 2 Decision 3

Model of Drug Traffickers based on BDI framework

Behavioral Models of Drug Traffickers

Behavior Models of Border Patrols

Environment Conditions:weather, terrains 3-Level Sensor Networks

Developed behavior models of drug traffickers and ground patrol agentstogether with environmental conditions will provide rich scenarios


Agent Models for Patrol Agent and Sensors

SensorsPatrol Agents


2.0- Pattern Processing Recognition and Prediction of objective, behavior and route patterns of

hostile targets

• Spatial Optimization Method


Emerging Challenges - Pattern Processing

0 0 1 0 …0 1 0 0 …2 0 0 0 …0 0 3 0 …

… … … … …

… … … … …

3D Grid Matrix with Categorized Valuese.g. 0: Unoccupied; 1: Hostile Crowd; 2: Friendly Crowd;3: Animals

Challenge: Computational complexity, modeling the interactions between targets.


3.0- Mission Control Resource allocation and Motion planning for controlling hostile targets

to terminate/mitigate their activities

Predicting Target’s Behavior

Optimized Sensors

Allocation

PersistentSurveillance


Challenge: risk assessment and uncertainty quantification

Emerging Challenges - Mission Control

Target Risk1 0.752 0.973 0.23… …

Target Risk Assessment

+

New ObservationUpdating

Target Pattern Prediction

Target1

Target2

Target3

Allocation Optimization


DDDAS in Modules: Target DRIAPPLICATIONAPPLICATION

3-D spatial model3-D spatial model

Heterogeneous models for both civilian and

hostile targets

Heterogeneous models for both civilian and

hostile targets

Multi-scale model for dynamics and BDI for

human decisions

Multi-scale model for dynamics and BDI for

human decisions

Model uncertainty for miss-classification


Sensors and vehicles are modeled as agentsSensors and vehicles are modeled as agents

Online camera motion strategies in 3D

Online camera motion strategies in 3D

Fuse high-level motion command and dynamic

environmental information

Fuse high-level motion command and dynamic

environmental information

ALGORITHMSALGORITHMS

Multi-scale model with more distributions to

model spatial, temporal and preference

Multi-scale model with more distributions to

model spatial, temporal and preference

Mixture of regressions analysis algorithm

Mixture of regressions analysis algorithm

Information fusion algorithm for target

Identification

Information fusion algorithm for target

Identification

Dynamic model parameter updating for

hostile targets

Dynamic model parameter updating for

hostile targets



Image and geometry registration algorithms Image and geometry

registration algorithms

Online motion planning methods using

estimated “earliest collision time” in 3D

Online motion planning methods using

estimated “earliest collision time” in 3D

MEASUREMENTMEASUREMENT

3-D data3-D data

City-size mid-resolution dynamic

data by Aerostat


data by Aerostat

Mid-range High-resolution dynamic

data by UAVs


data by UAVs

Adjacent high-resolution dynamic

data by UGVs


data by UGVs

Status of sensors and vehicles

measured


measured

SOFTWARESOFTWARE

Bayesian updating software module

for location, dynamic and

sensor/vehicle status modeling




Heterogeneous model estimation software module


Extension of Repast Simphony®-based HIL simulator with

more sensors, a richer set of sensory and commands data



Detailed models of agents (e.g. sensors) in a Physics-based

simulation software (e.g. Gazebo, MAK,

AGI, etc.)



AGI, etc.)


DDDAS in Modules: Pattern ProcessingAPPLICATIONAPPLICATION

3-D spatial model3-D spatial model

Assume hostile traffickers detected

Assume hostile traffickers detected

Heterogeneous models for location,

dynamics and preference pattern

recognition

Heterogeneous models for location,

dynamics and preference pattern

recognition

Model uncertainty for pattern

misrecognition

Model uncertainty for pattern

misrecognition

Impacts of sensors/vehicles status considered

Impacts of sensors/vehicles status considered


Patterns extracted as statistics

extracted from algorithm

Patterns extracted as statistics

extracted from algorithm

Mixture of regressions

analysis algorithm

Mixture of regressions

analysis algorithm

Information fusion algorithm gives

statistical patterns

Information fusion algorithm gives

statistical patterns

Pattern extraction and recognition

algorithm will be developed

Pattern extraction and recognition

algorithm will be developed

Algorithm to update prior

knowledge on patterns will be

developed

Algorithm to update prior

knowledge on patterns will be

developed


3-D data3-D data


data by Aerostat


data by Aerostat


data by UAVs


data by UAVs


data by UGVs


data by UGVs


measured


measured

SOFTWARESOFTWARE















AGI, etc.)



AGI, etc.)


DDDAS in Modules: Mission ControlAPPLICATIONAPPLICATION

3-D pattern model

3-D pattern model

Heterogeneous models

Heterogeneous models

Controllability is estimated via

repeated simulations

Controllability is estimated via

repeated simulations

Augmenting the Surface Level’s

vision

Augmenting the Surface Level’s

vision


Control index considered for 3-D

optimality

Control index considered for 3-D

optimality

Algorithm uncertainty caused

by both model estimation

uncertainty and sensor/vehicle

uncertainty

Algorithm uncertainty caused

by both model estimation

uncertainty and sensor/vehicle

uncertainty

Algorithm updated considering agent dynamics, pattern-

shifting and sensor/vehicle

status

Algorithm updated considering agent dynamics, pattern-

shifting and sensor/vehicle

status


3-D data3-D data


data by Aerostat


data by Aerostat


data by UAVs


data by UAVs


data by UGVs


data by UGVs


measured


measured

SOFTWARESOFTWARE















AGI, etc.)



AGI, etc.)


Acknowledgements

PIs: Young-Jun Son1, Jian Liu1, Jyh-Ming Lien2

Students (Previous Project): A. Khaleghi1, D. Xu1, S. Minaeian1, Y. Yuan1, M. Li1, and C. Vo2

Students (New Project): S. Minaeian1, Y. Yuan1, S. Lee1, and J. Han1

1Systems and Industrial Engineering, University of Arizona2Computer Science, George Mason University

PI Contacts:[email protected]; [email protected]; [email protected]; 1-703-993-9546

Sponsor: Air Force Office of Scientific Research

FA9550-12-1-0238 (DDDAS); 15RT1016 (New)Program Manager: Dr. Frederica Darema

A Hierarchical and Scalable Situation Awareness System … · A Hierarchical and Scalable Situation...

Documents

Transcript of A Hierarchical and Scalable Situation Awareness System … · A Hierarchical and Scalable Situation...