Real-Time and Big Data Analysis for Defense...Bringing real-time and big data capabilities to your...
Transcript of Real-Time and Big Data Analysis for Defense...Bringing real-time and big data capabilities to your...
John Fry, Kenyon Huber-Wilker
Real-Time and Big Data Analysis for
Defense and Intelligence Workflows
things that move… things that “just happen”…
• airplanes
• vehicles
• animals
• storms
• crimes
• lightning
• accidents
• tweets
Real-Time GIS – Types of observations and data
stationary sensors…
• water gauges
• weather stations
• air quality sensors
• device temperature
• ships
• satellites
• trains
• people
What about storing, visualizing, and analyzing those features?
What about storing, visualizing, and analyzing those features?
Public Safetypolice fire
surveillance
Public Healthhospitals ambulances
Transitbuses taxis rail
trains crowds
Connected Carsautonomous driving traffic conditions holes
parking meters road conditions slippery areas
network improvements
Energy Usageelectricity gas smart meters
City Workerssanitation snow plows
Airportsflight status queues
plane location runway status
Buildingslighting hvac
occupancy counts
Internet of your Things
Weatherwarnings earthquakes
precipitation icy conditions
Peoplehealth monitoring
social activityTelecommunications
cell phone signals dropped calls
enabling geospatial insights with your IoT ArcGIS and IoT
Environmentnoise co2 nitrates
gases temperature humidity
atmospheric pressure radiationpesticides electromagnetic feedback
rain gauges water level gaugeswater quality air quality
Apps
DesktopAPIs
with real-time & big data capabilities
ArcGIS Enterprise
Ready to Use Real-Time ServicesArcGIS Living Atlas provides real-time content
https://www.esri.com/arcgis-blog/products/real-time/real-time/putting-the-live-in-living-atlas-of-the-world/
USA Weather Watches and Warnings
USA Short-Term Weather Warnings
USA Storm Reports
Current Weather and Wind Station Data
Active HurricanesRecent Hurricane
Satellite (MODIS) Thermal Hotspots and Fire Activity
National Weather Service Wind Forecast
National Weather Service Wind Gust Forecast
National Weather Service Wind Direction Forecast
National Weather Service Precipitation Forecast
National Weather Service Snowfall Forecast
National Weather Service Ice Forecast
National Weather Service Smoke ForecastRecent Weather Radar Imagery
And more…
• Weather and Disaster Live Feeds have been updated
• No subscription needed
• Shared as feature layers
• Auto updates in web map
John Fry
Ready to use live
services
Field Operations with ArcGIS
Field Operations
Workforce
Navigator
ArcGIS ProOperations Dashboard
Tracker
Explorer
Drone2Map
Survey123
Collector
UNDERSTAND
QuickCapture
COLLECT
WorkforceNavigator
Collector
Survey123Explorer
Tracker
G
QuickCapture
. . . Empowering Mobile Users
Tracker
Workforce
Collector
Navigator
Explorer
Survey123
QuickCapture
Integrated Mobile Apps for the Field
Demo – Tracker
and Operations
Dashboard
ArcGIS Enterprisewith real-time & big data capabilities
Apps
DesktopAPIs
analytics storage
visualization
ingestion
analytics
GeoAnalytics
Server
GeoAnalytics
Desktop
toolbox
Configurable inputs for any type of data❖ Ingest Data:
GeoEvent Server – Key Capability #1
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Bringing real-time and big data capabilities to your ArcGIS Enterprise
Filter:
Blue / Orange
▪ Identify event records with specified attribute values
❖ Apply real-time analytics: Filtering
GeoEvent Server – Key Capability #2
GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Bringing real-time and big data capabilities to your ArcGIS Enterprise
▪ Identify event records with specified attribute values
▪ Determine spatial relationships with established geofences
❖ Apply real-time analytics: Filtering
GeoEvent Server – Key Capability #2
geofence
outside
inside
outside
geofence
disjoint crosses
GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Bringing real-time and big data capabilities to your ArcGIS Enterprise
✓ Field Calculator
✓ Field Mapper
✓ Buffer Creator
✓ GeoTagger
✓ Incident Detector
✓ Track Gap Detector
…
GeoEvent Server – Key Capability #2
▪ Configure and use over two dozen types of processors
❖ Apply real-time analytics: Processing
GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Bringing real-time and big data capabilities to your ArcGIS Enterprise
GeoEvent Server – Key Capability #2
▪ Configure and use over two dozen types of processors
❖ Apply real-time analytics: Processing
✓ Field Calculator
✓ Field Mapper
✓ Buffer Creator
✓ GeoTagger
✓ Incident Detector
✓ Track Gap Detector
…GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Bringing real-time and big data capabilities to your ArcGIS Enterprise
▪ Configurable outputs enable processed events to bestored in a traditional relational geodatabase
▪ When data is received at a high volume, high velocity,or is accumulated over time, a spatiotemporal big datastore is an available noSQL option for enterprise storage
❖ Store data as feature records in a geodatabase
GeoEvent Server – Key Capability #3
GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Bringing real-time and big data capabilities to your ArcGIS Enterprise
▪ Relational geodatabase feature services supporttraditional RESTful web applications and APIs
▪ Big data map and feature services support on-the-flyfeature aggregation
❖ Visualize feature records
GeoEvent Server – Key Capability #4
GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Bringing real-time and big data capabilities to your ArcGIS Enterprise
▪ Relational geodatabase feature services supporttraditional RESTful web applications and APIs
▪ Big data map and feature services support on-the-flyfeature aggregation
▪ Stream Services
❖ Visualize feature records
DesktopApps and
APIs
ArcGISData Store
ArcGISEnterprise
GeoEventServer
GeoEvent Server – Key Capability #4
GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Bringing real-time and big data capabilities to your ArcGIS Enterprise
▪ Configurable outputs provide the capability to format ane-mail or SMS text message
▪ How you choose to disseminate processed event recordsis entirely up to you
❖ Notify stakeholders about detected patterns of interest
GeoEvent Server – Key Capability #5
!
GeoEvent Server
server role
Spatiotemporal big
data store
Apps
Desktop
APIs
Consuming Real-Time Data
Stream Layer
Map LayerFeature Layer
• Stream layers in apps subscribe to stream services to immediately visualize observations
- does not require storage, low latency, no playback
• Map & Features layers in apps periodically poll to visualize most current observations
- backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS)
- history can be retrieved & queried for playback
ArcGIS
Enterprise
Stream Service
Send Features to a Stream Service subscribe (push)
polling (pull)
Map Service
Feature
ServiceGeoAnalytics
Server
Add a Feature to a BDS
Update a Feature in a BDS
spatiotemporal
big data store
Add or Update a Feature
EGDBGeoEvent
Server
Apps
Desktop
APIs
John Fry
Land Operations
Land Operations
• Use Case:
- My organization needs to understand the location of blue forces, red forces, and intelligence
collection platforms.
• Workflow:
- Geoevent:
- Input – Simulation file for military units
- Output – Update a feature service, stream service
- GeoEvent Service
- Ingest feed of military units and receive alerts when in a target area
- Ingest FMV asset feed and create frame and line and alert when
over a target position
- Operations Dashboard
- List widgets, Map Widgets
LandOps GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
LandOps GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
LandOps GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
LandOps GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
John Fry
Satellite Tracking
Satellite Tracking
• Use Case:
- My organization needs to understand the location of different platforms, what type they are, and
whether they pose an intelligence risk to my assets.
• Workflow:
- Geoevent:
- Input – https://n2yo.com/api/ satellite JSON feed
- Output – Update a feature service
- GeoEvent Service
- See location of what is above center of USA
- Request for Yaogan Series satellite across the globe
and alert whether they are approaching a US military bases
- Operations Dashboard
- List widgets, Bar Charts, Embedded Content, Indicator Widgets
John Fry
Satellite Tracking
Satellite GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
Satellite GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
Satellite GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
Satellite GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
Satellite GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
Satellite GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
Satellite GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
Satellite GeoEvent Workflow – Key Components
Inputs OutputsFilters and Processors
Field Mapper
John Fry
Maritime Big Data
Maritime Big Data
• Use Case:
- My organization needs to ingest ship Automatic Identification System (AIS) position data
and store it
• Workflow:
- Geoevent:
- Input – Spire’s AIS - https://www.spire.com/en
- Output –
- Add a feature to spatiotemporal big data store
- GeoEvent Service
- Ingest SPIRE and add these positions to a spatiotemporal big data store
- WebAppBuilder:
- Query data
John Fry
Maritime Big Data
ArcGIS Enterprisewith real-time & big data capabilities
Apps
DesktopAPIs
analytics storage
visualization
ingestion
analytics
GeoAnalytics
Server
GeoAnalytics
Desktop
toolbox
GeoAnalytics parallelizes computing (Apache
Spark) to quickly analyze large amounts of
vector and tabular data
A collection of analysis tools to identify
patterns, relationships, anomalies and
incidents in large amounts of data across
space and time
What is GeoAnalytics?
GeoAnalytics is available through...
• ArcGIS Pro (2.4+) – GeoAnalytics Desktop Tool
-ArcGIS Pro Advanced license
• GeoAnalytics Server (10.5+) – use 1 or 3 machines to
distribute analysis
-Access tools in:
-Map Viewer
-ArcGIS Pro
-ArcGIS REST API
-ArcGIS API for Python
Summarize Data
Find Locations
Analyze Patterns
Use Proximity
Manage Data
GeoAnalytics Server Tools
Data Enrichment
Tool availability between clients may have minor differences (Map Viewer, Pro, REST API, Python API)
Summarize Data
Find Locations
Analyze Patterns
Use Proximity
Manage Data
Data Enrichment
GeoAnalytics Desktop Tools (ArcGIS Pro 2.4)
Summarize Data
Powerful Tool – Join Features
Spatial
• Intersects
• Equals• Near
• Contains• Within• Touches
• Crosses• Overlaps
Temporal
• Meets
• Met by• Overlaps
• Overlapped by• During• Contains
• Equals• Finishes
• Finished by• Starts• Started by
• Intersects• Near
Attribute
Features are matched
when the field values
in the join layer are
equal to field values in
the target layer.
Near spatial and temporal relationships require a distance/length of time to be specified. For more information see:
https://pro.arcgis.com/en/pro-app/tool-reference/big-data-analytics/spatial-relationships-with-big-data.htm
Joins in GeoAnalytics Server
When to use Desktop or Server for GeoAnalytics?
• Use GeoAnalytics Desktop when you want to:
- Process local data (from files, databases) faster than before on your own desktop machine
- Prototyping workflows you want to use with GeoAnalytics Server
• Use GeoAnalytics Server when you want to:
- Bring big data analysis to your entire organization
- Leverage the power of one or multiple server machines
- Connect to external big data storage and existing web layers
- Extend using custom analysis
John Fry
GeoAnalytics
Desktop toolbox
Kenyon Huber-Wilker
Cell Phone
Analysis
ArcGIS Enterprisewith real-time & big data capabilities
Apps
DesktopAPIs
analytics storage
visualization
ingestion
analytics
GeoAnalytics
Server
GeoAnalytics
Desktop
toolbox
Guides / Resources
Self-paced training and introductions to GeoEvent Server
Real-Time GIS: GeoEvent Server
• Step-by-Step Tutorials, free to download
• http://enterprise.arcgis.com/en/geoevent/latest/
get-started/geoevent-server-tutorials.htm
- Introduction
- Stream services
- Spatiotemporal Big Data Store
- Notifications
• Blogs and discussions on the forum
- http://links.esri.com/geoevent-forum
• Video recordings of technical workshops
- http://www.esri.com/videos
• GeoEvent Best Practices
Guides / Resources
GeoAnalytics
• Tutorial: Run a GeoAnalytics tool
• Spark-Powered Analysis with GeoAnalytics
Desktop Tools vs. GeoAnalytics Server
• GeoAnalytics Server Resource List
• What is GeoAnalytics Server?
• Detecting Incidents with Ozone Data
• Water Leaks blogs
• Data in GeoAnalytics Server
• Extend your Analysis with GeoAnalytics Server
and Spark (Run Python Script
Questions?
Print Your Certificate of Attendance
Print Stations Located in 150 Concourse Lobby
Tuesday12:30 pm – 6:30 pm
Expo
Hall B
5:15 pm – 6:30 pm
Expo Social
Hall B
Wednesday10:45 am – 5:15 pm
Expo
Hall B
6:30 pm – 9:30 pm
Networking Reception
Smithsonian National Museum
of Natural History
Download the Esri
Events app and find your event
Select the session
you attended
Scroll down to
“Survey”
Log in to access the
survey
Complete the survey
and select “Submit”
Please Share Your Feedback in the App