Real-time and Big Data Analysis for Defense and ...€¦ · When data is received at a high volume,...

Real-Time and Big Data Analysis for Defense and Intelligence WorkflowsCurt HammillJohn FryTyson Quink

things that move… things that “just happen”…

• airplanes

• vehicles

• animals

• storms

• crimes

• lightning

• accidents

• tweets

Real-Time GIS – Types of observations and data

stationary sensors…

• water gauges

• weather stations

• air quality sensors

• device temperature

• ships

• satellites

• trains

• people

What about storing, visualizing, and analyzing those features?

Public Safetypolice fire

surveillance

Public Healthhospitals ambulances

Transitbuses taxis rail

trains crowds

Connected Carsautonomous driving traffic conditions holes

parking meters road conditions slippery areas network improvements

Energy Usageelectricity gas smart meters

City Workerssanitation snow plows

Airportsflight status queues

plane location runway status

Buildingslighting hvac

occupancy counts

Internet of your Things

Weatherwarnings earthquakes

precipitation icy conditions

Peoplehealth monitoring

social activityTelecommunications

cell phone signals dropped calls

enabling geospatial insights with your IoT ArcGIS and IoT

Environmentnoise co2 nitrates

gases temperature humidityatmospheric pressure radiation

pesticides electromagnetic feedbackrain gauges water level gaugeswater quality air quality

Apps

DesktopAPIs

with real-time & big data capabilities

ArcGIS Enterprise

ArcGIS Enterprisewith real-time & big data capabilities

DesktopWeb Device

live & historicaggregates & features

map & feature servicestream service

live features

ArcGISEnterprise

GeoEventServer

analytics

GeoAnalyticsServer

spatiotemporalbig data store

storage analytics

visualization

ingestion

actuation

Bringing real-time and big data capabilities to your ArcGIS Enterprise

GeoEvent Serverserver role

spatiotemporal bigdata store

Platform Location Ingest Data:

GeoEvent Server – Key Capability #1




Sensors and Instruments Ingest Data:





Weather Monitoring Ingest Data:





National Security Data Feeds Ingest Data:





Industrial, Utilities, Facility Ingest Data:





Configurable inputs for any type of data Ingest Data:



Filter:Blue / Orange

Identify event records with specified attribute values Apply real-time analytics: Filtering





Identify event records with specified attribute values Determine spatial relationships with established geofences

Apply real-time analytics: Filtering


geofence

outside

inside

outside

geofence

disjoint crosses




Field Calculator

Field Mapper

Buffer Creator

GeoTagger

Incident Detector

Track Gap Detector

…


Configure and use over two dozen types of processors Apply real-time analytics: Processing





Configure and use over two dozen types of processors Apply real-time analytics: Processing



Field Calculator

Field Mapper

Buffer Creator

GeoTagger

Incident Detector

Track Gap Detector

…


Configurable outputs enable processed events to bestored in a traditional relational geodatabase

When data is received at a high volume, high velocity,or is accumulated over time, a spatiotemporal big datastore is an available noSQL option for enterprise storage

Store data as feature records in a geodatabase





Relational geodatabase feature services supporttraditional RESTful web applications and APIs

Big data map and feature services support on-the-flyfeature aggregation

Visualize feature records





Relational geodatabase feature services supporttraditional RESTful web applications and APIs

Big data map and feature services support on-the-flyfeature aggregation

Stream Services

Visualize feature records





Configurable outputs (email, messages, push) How you choose to disseminate processed event records

is entirely up to you

Notify commanders and staff about detected patterns of interest


!



Consuming Real-Time Data

Stream LayerMap Layer

Feature Layer

• Stream layers in apps subscribe to stream services to immediately visualize observations- does not require storage, low latency, no playback

• Map & Features layers in apps periodically poll to visualize most current observations- backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS)- history can be retrieved & queried for playback

ArcGISEnterprise

Stream Service

Send Features to a Stream Service subscribe (push)

polling (pull)

Map ServiceFeature ServiceGeoAnalytics

Server

Add a Feature to a BDSUpdate a Feature in a BDS


Add or Update a Feature

EGDBGeoEvent

ServerApps

Desktop

APIs

Copyright © Esri. All rights reserved.

Operations Dashboard for ArcGIS

• Monitor assets, events, or activities for at-a-glance decision making

• Configurable user experience

• Command center in the web

• Integrated into the enterprise

John FrySatellite Tracking

Satellite Tracking

• Use Case:- My organization needs to understand the location of different platforms, what type they are,

and whether they pose an intelligence risk to my assets.

• Workflow:- Geoevent:

- Input – https://n2yo.com/api/ satellite JSON feed- Output – Update a feature service- GeoEvent Service

- See location of what is above center of USA- Request for Yaogan Series satellite across the globeand alert whether they are approaching a US military bases

- Operations Dashboard- List widgets, Bar Charts, Embedded Content, Indicator Widgets

https://n2yo.com/api/

John FrySatellite Tracking

Satellite GeoEvent Workflow – Key ComponentsInput OutputField Mapper

Field Calculator -Height

Field Calculator -Geometry

Field Enricher

John FryAircraft Watchlist

Aircraft Watchlist Workflow

• Use Case:- My organization needs to understand the location of a

platform and enrich that location with watchlist information.


- Input – CSV and TCP input- Output – Update a feature service- GeoEvent Service

- Aircraft locations is enriched by filling out a geoform. A geofence around DC is used as an incident detector.

- Operations Dashboard- List widgets, Map Widget, Embedded Content

John FryAircraft Watchlist

Aircraft Workflow Key Components Input

Processors and filters

Field Enricher

Output

John Fry

Maritime Situational Awareness

Maritime Domain Awareness

• Use Case:- My organization needs to understand the location of vessels at sea and whether they are in

the area of a threat, or, pose a threat to operations.


- Input – Spire’s AIS- Output –

- Add a feature to Spatiotemporal big data store- Update an alert feature in hazard areas

- GeoEvent Service- Ingest SPIRE and add these positions to a data store to store many records, while also getting alerts when

ships enter NGA NAVAL Warning areas.

- Operations Dashboard- List widgets, Bar Charts, Indicator Widgets, Map Widgets

Maritime – GeoEvent Service

Input Output to BDS

Field Mapper

Filter Merchant

Vessels and Speed < 5

Filter Vessels in NAVAREA

Filter by Geofence not in EEZ

Calculate Field

Calculate Field

Field Mapper

Output alert feature service

What is GeoAnalytics Server?

• ArcGIS has a new way of processing spatiotemporal data that is designed for big data by leveraging distributed analytics and storage- Delivered as tools in ArcGIS GeoAnalytics Server

and ArcGIS Pro- Familiar workflows and UX in the portal Map Viewer

and ArcGIS Pro- Works with your existing GIS data and tabular data

– not just big data- Designed to perform both spatial and temporal

analysis

Aggregate PointsJoin FeaturesReconstruct TracksSummarize AttributesSummarize Within

Summarize Data

Find Locations

Analyze Patterns

Use Proximity

Manage DataFind Similar Locations

Calculate DensityCreate Space Time CubeFind Hot Spots

Create Buffers

Copy to Data Store

GeoAnalytics Server | Rich Collection of Analysis Tools

GeoAnalytics Server: analytic capabilitiesBatch analysis

Analyze Patterns

Use Proximity

Summarize Data

Calculate DensityFind Hot Spots

Create Space Time Cube

Join FeaturesAggregate PointsSummarize Within

Summarize AttributesReconstruct Tracks

Detect IncidentsFind Similar Locations

Geocode Locations

Create Buffers

Find Locations

“I want to…”

Manage Data Calculate FieldCopy To Data Store

GeoAnalyticsServer

summarize dataBatch analysis• Aggregate Points:

- “How does the spatial distribution of vehicle collisions change over time?”

- “What zip codes have the highest count of crimes incidents?”

- ”Where are there the most power outages?”

- ”What does my data look like?”


- Input 1: Points, Input 2: Polygons (or generate bins)

- Output: Polygons

- Aggregates point data into either a square grid, a hexagonal grid, or user-supplied polygon features

- Users can choose to aggregate either spatially or spatiotemporally

- A count of points is returned for each grid cell (bins), in addition to optional attribute field statistics


Methods of Aggregation

Spatial, into polygons:

Spatial, into bins:


Methods of Aggregation

Spatiotemporal, into bins:

Spatiotemporal, into polygons:

Spatial, into polygons:

Spatial, into bins:

summarize dataBatch analysis• Join Features:

- “Which crime events occurred near sporting events, spatially & temporally?”

- “Which bodies of water intersect cities with populations greater than one million people?”

- “Which traffic jams occurred because of car accidents?”


- Input 1: Points, Polygons or Lines, Input 2: Points, Polygons or Lines

- Output: Points, Polygons or Lines

- Joins attributes from one layer to another based on spatial, temporal & attribute relationships, or any combination of those relationships.


Spatial

• Intersects• Equals• Near• Contains• Within• Touches• Crosses• Overlaps

Temporal

• Meets• Met by• Overlaps• Overlapped by• During• Contains• Equals• Finishes• Finished by• Starts• Started by• Intersects• Near

AttributeFeatures are matched when the field values in the join layer are equal to field values in the target layer.

Near spatial and temporal relationships require a distance/length of time to be specified. For more information see: https://pro.arcgis.com/en/pro-app/tool-reference/big-data-analytics/spatial-relationships-with-big-data.htm

Joins in GeoAnalytics Server

https://pro.arcgis.com/en/pro-app/tool-reference/big-data-analytics/spatial-relationships-with-big-data.htm

Tyson Quink

Maritime Historic Patterns

Tyson QuinkGlobal Analysis

ArcGIS Enterprisewith real-time & big data capabilities

DesktopWeb Device

live & historicaggregates & features

map & feature servicestream service

live features

ArcGISEnterprise

GeoEventServer

analytics

GeoAnalyticsServer


storage analytics

visualization

ingestion

actuation

Guides / Resources

Self-paced training and introductions to GeoEvent Server

Real-Time GIS: GeoEvent Server

• Step-by-Step Tutorials, free to download• http://enterprise.arcgis.com/en/geoevent/latest/

get-started/geoevent-server-tutorials.htm- Introduction- Stream services- Spatiotemporal Big Data Store- Notifications

• Blogs and discussions on the forum- http://links.esri.com/geoevent-forum

• Video recordings of technical workshops- http://www.esri.com/videos

• GeoEvent Best Practices

http://www.esri.com/videoshttp://proceedings.esri.com/library/userconf/proc17/tech-workshops/tw_356-376.pdf

GeoEvent Serverbest practices tutorial for multi-machine site deployment

• Available Now: http://links.esri.com/geoevent-multiplemachine

Print Your Certificate of AttendancePrint Stations Located at L Street Bridge

Tuesday Wednesday12:30 pm – 6:30 pm GIS Solutions Expo Hall D

5:15 pm – 6:30 pm GIS Solutions Expo SocialHall D

10:45 am – 5:15 pm GIS Solutions Expo Hall D

6:30 pm – 9:00 pm Networking ReceptionNational Building Museum

Please Take Our Survey on the AppDownload the Esri Events app and find your event

Select the session you attended

Scroll down to find the feedback section

Complete answersand select “Submit”

Real-Time and Big Data Analysis for Defense and Intelligence WorkflowsSlide Number 2Slide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Operations Dashboard for ArcGISSatellite TrackingSatellite TrackingSatellite TrackingSatellite GeoEvent Workflow – Key Components�Aircraft WatchlistAircraft Watchlist WorkflowAircraft WatchlistAircraft Workflow Key Components �Maritime Situational AwarenessMaritime Domain AwarenessMaritime – GeoEvent ServiceWhat is GeoAnalytics Server?Slide Number 35Slide Number 36Slide Number 37Slide Number 38Slide Number 39Slide Number 40Slide Number 41Slide Number 42Slide Number 43Maritime Historic PatternsGlobal AnalysisSlide Number 47Guides / ResourcesSlide Number 49Slide Number 50Slide Number 51Slide Number 52

Real-time and Big Data Analysis for Defense and ...€¦ · When data is received at a high volume,...

Documents

Transcript of Real-time and Big Data Analysis for Defense and ...€¦ · When data is received at a high volume,...