DARPA Insight Industry Day Briefing

Approved for Public Release, Distribution Unlimited

InsightIndustry DayDr. Michael PagelsProgram Manager

Information Innovation Office

National Rural Electric Cooperative Association Conference Center 21 & 22 September 2010


Problem: Limited Ability to Automatically Exploit and Cross-Cue Multi-INT Sources

2

Checkpoint: Suspect 1 in Vehicle A passed

HUMINT: Verifies Suspect 3 seen leaving residence

Activity Alert: Vehicle A stops, Suspect 3 enters

HUMINT Report: Suspect 2 interviewed

Suspects converge to same location

1

38

5

6

7 HUMINT: Conflicting information– Suspect 3 seen in Paris

Insight: Cross-cue sensor, initiate tracks, associate track to report

Insight: Associate tracks and threat network information,assess state uncertainties

Insight: Cross-cue, initiate tracks, associate track to report

Insight: Associate tracks, cue video activity processing, alert operator

Insight: Update uncertaintiesand alert

Insight: Update uncertainties and alert, cue source analyst for verification

Insight: Predicts track loss, cross-cue alternative sensor

2

Insight: Stimulate device to update state uncertainties

4

0

Post-Checkpoint: vehicle driving

Comm Device Emission

Operational example:High Value Individual (HVI)tracking with if-and-thenconstraints

Watchlist posted andaccepted by analyst

Insight: Add watchlistto active tasks, initiate data search


Existing and planned systems connect multiple real-time sources to tactical operations centers for largely manual exploitation and cross-cueing via chat-based operator interaction

• No automated correlation and uncertainty management

• Limits the number of threats to a fraction of the number of operators

• Limits scenario complexity to less than sensors observe

• No automated collection task assessment and management

• Limits the ability to prosecute agile threats

• Limits response times amid rapidly changing conditions

• No integrated human-machine reasoning

• Limits the ability to exploit operator knowledge of context

• Limits operator ability to reason over complex data

• Warfighters cannot select best asset mix to employ for a given mission

• Assets are underutilized or unavailable


3



4

Capability Objective Context and Mission Adaptability

Human Machine Interaction

HVI tracking withmultiple conditional constraints

Detect and track multiple HVIs; satisfy multiple alertingconstraints

• Deliberate decision making and collaboration

• Identity confusion management

• Activity analysis• Resource management

Long term HVI tracking

Identify and track threat dismounts and vehicles through clutter and obscuration

• Suburban domain• Rural domain• Rural-to-Suburban-to-Urban

• Track stitching• Track correlation• Resource management

Estimating the threat network

Identify threat relationships, roles, and associated physical infrastructure

• Compound area networks• Urban area networks• Dispersed rural networks

• Threat network hypothesis generation

• Resource management

Behavioral modeling, prediction, and anomaly detection

Estimate normal patterns of activity on roads, along borders, and in areas friendly forces will be operating

Detect deviations from normalcy

• Mobile force overwatch• Site monitoring• Road monitoring• Border monitoring

• Normalcy estimation• Pattern detection• Pattern prediction• Anomaly detection• Resource management


Solution: Next Generation ISR Exploitation System, Evaluated At NTC Physical Test Bed

5

Development Incubator• Collected, simulated and

processed data• Variable resolution• Multiple security levels• Support infrastructure

Virtual Environment • Simulated sensor data• Simulated world of various threats,

terrains, and terrain features• Augmented with real-world

collected data

Physical Test Bed• National Training Center• Realistic operational conditions

• Increase the number of participants• Lower the cost of entry for participants

• System evaluation under realistic operational conditions

• Supports Service transition

Theater

Insight Test Bed

Adaptive multisource exploitation & resource management tools

Unified all-source ISR workstation

Algorithm Command, Control & Data Archive

Insight Exploitation & Resource Management (E&RM) System• Build model-based correlation, behavioral modeling, and threat network analysis tools to automatically combine

data across sources and manage uncertainty.• Build resource management tools to identify collection opportunities, and enable efficient use of multi-INT

sensors and platforms across missions.• Build a unified data management and processing environment to support data and processing normalization,

and enable rapid integration of new data sources, exploitation algorithms, and analysis methodologies.• Integrate human and machine processing, including visualization, hypothesis manipulation, crowd-sourcing,

and on-line learning.

• Test multiple concepts of operation (CONOPs)• Test alternative sensor mixes• Test multiple exploitation algorithm solutions


Notional

Insight E&RM System:Three components

6

TrackMultiple Futures

Prediction

Histories

Cues

ClassifyInterpretation

Registered Data Layers:• Open Source (OSINT)• Law Enforcement• Commercial Sources• IMINT (Imagery, Video, Wide

Area Motion Imagery (WAMI))• GMTI/VMTI• SIGINT• COMINT• MASINT• CI/HUMINT• Soldier Information• SIGACTS• Ground Based Sensors

Threat Description Information Needs

Known Threats

Forensic Studies

New Behaviors and Features

SearchExtraction &

Link DiscoveryThreat Description

Anomaly RecognitionTracksID of IndividualsBehavior Patterns Node and Link AnalysisRolesBackground/Environment

Analysts

1: Analysis and Resource Management System

Threat and Background Models

• Leadership• Mission Planning• Bombmakers• Supply Chain• Financers• Recruiting• IED Emplacers• GIS

Multi-Mode Collection (Sensing)

Actionable Intelligence

Information Needs

Archive for Forensic Studies and Forensic Analysis Reach back

Resource Management

Collection Tasks

2: Unified All-sourceISR Human-Machine Interface

3: Integration, Processing and Data Archive Environment

Notional Architecture

http://www.mccoy.army.mil/vtriad_online/02242006/images/


Insight E&RM System Component 1: Analysis & resource management system

Develop information exploitation analysis tools which automatically combine and process data across multiple sources to fulfill multiple ISR missions

• model-based correlation• behavioral modeling• threat network analysis

Develop collection and sensor management tools smart enough to know when and how to cooperatively prosecute missions and targets

Provide the foundational research for enhancing algorithmic performance by visualization of intermediate results, human intervention in hypothesis resolution, and interactive on-line learning

Technology push towards open, modular systemsUnbundle exploitation system development stovepipes

into minimal components to reduce the time and cost to develop future ISR systems

7

Development Incubator

Physical Test Bed

Virtual Environment

User interaction drives the automated exploitation process


Insight E&RM SystemComponent 2: Unified all-source ISR human-machine interface

Modern human-computer interface (HCI), which acts as the visual and procedural gateway to Insight’s management and exploitation capabilities

Provide foundational research into enhancing effective interaction between human and E&RM system

Provide access to exploitation tools and products over a wide range of interface devices to maximize multiple echelon coverage

Adaptable, customizable, context-aware, unified interface capable of supporting dynamic, interactive missions

• Social network leveraged to maximize collaboration• “Push” technology leveraged for information discovery• Adapts to task and mission focus covering all training,

planning and exploitation phases• Plug-in architecture supporting rapid integration and

evaluation of innovative HCI technologies• Standards-based implementation supporting dissemination

of HCI technologies to other systems and domains

8

Insight Analyst

User interaction drives the automated exploitation process

User interaction with both real and virtual data supporting multiple exploitation phases

Insight Workstation


Insight E&RM System Component 3: Integration, Processing and Data Archive Environment

Architected to maintain an adaptive and agile user experience by rapidly integrating new algorithms via a standardized application programming interface

Manages and monitors the mapping of the E&RM system’s dynamic algorithms on to a distributed heterogeneous execution environment of computing and visualization resources

Captures the provenance of all products in the processing chain to explain the reasons behind algorithmic exploitation results

Provides the unified data source for all processing• Common standards-based, spatial, temporal and error

measure registration techniques to unify layers across pixel, text and other data representations

• Real sensor output and virtual/simulated data coexist• Leverage existing diverse COTS database technologies

9

TrackMultiple Futures

Prediction

Histories

Cues

ClassifyInterpretation

New Behaviors and Features

SearchExtraction &

Link Discovery

Threat Description

Threat and Background Models• Leadership• Mission Planning• Bombmakers• Supply Chain• Financers• Recruiting• IED Emplacers• GIS


Information Needs

Resource Management

Collection Tasks

Distributed Heterogeneous Execution EnvironmentDiverse platforms required for diverse missions


Global ISR solution has two elements

Development Incubator• Collected, simulated and

processed data• Variable resolution• Multiple security levels• Support infrastructure

Virtual Environment • Simulated sensor data• Simulated world of various threats,

terrains, and terrain features• Augmented with real-world

collected data

Physical Test Bed• National Training Center• Realistic operational conditions

• Increase the number of participants• Lower the cost of entry for participants

• System evaluation under realistic operational conditions

• Supports Service transition

Theater

10

Insight Test Bed

Adaptive multisource exploitation & resource management tools

Unified all-source ISR workstation

Algorithm Command, Control & Data Archive

Insight Exploitation & Resource Management (E&RM) System

Solution: Next Generation ISR Exploitation System, Evaluated At NTC Physical Test Bed

• Build model-based correlation, behavioral modeling, and threat network analysis tools to automatically combine data across sources and manage uncertainty.

• Build resource management tools to identify collection opportunities, and enable efficient use of multi-INT sensors and platforms across missions.

• Build a unified data management and processing environment to support data and processing normalization, and enable rapid integration of new data sources, exploitation algorithms, and analysis methodologies.

• Integrate human and machine processing, including visualization, hypothesis manipulation, crowd-sourcing,and on-line learning.

• Test multiple concepts of operation (CONOPs)• Test alternative sensor mixes• Test multiple exploitation algorithm solutions


Insight Test Bed Component 1: Development incubator

Technology clearinghouse supporting development and rapid evaluation of new capabilities for inclusion in Insight’s E&RM System

• Provide access to technical and military experts• Provide operationally relevant mission descriptions and

context• Incremental evaluation utilizing historical data, virtual

environment and physical test bedProvides an unclassified data warehouse that lowers

the cost of participation via system evaluation and maximizes analysis of trade-offs for innovative development

• Generates “dialable” individualized views into the data• Captures processed data necessary for component

development and evaluationEnables leveraging of other funding opportunities

which broadens participation to non-traditional performers

Validates Insight’s Measure of Performance (MOP) development and evaluation among performers

Coordinates transition outreach and demonstrations

11

Registered Data Layers:• Open Source (OSINT)• Law Enforcement• Commercial Sources• IMINT (Imagery, Video, Wide Area Motion

Imagery)• GMTI/VMTI• SIGINT• COMINT• MASINT• CI/HUMINT• Soldier Information• SIGACTS• Ground-based Sensors


Collection Tasks

Development Incubator


Insight Test Bed Component 2: Virtual environment

Enables evaluation of ISR exploitation systems using simulated sensor data, augmented with real-world collected data, within a simulated world of various threats, terrains and terrain features.

• Test on sequestered data or data at higher classification levels not available in development incubator

• Test with data from simulated sensor and threat operating conditions beyond those in development incubator

• Assess system-level performance impacts of novel component-level algorithms

Enables system evaluation within multiple CONOPs, alternative sensor mixes, and multiple exploitation algorithm solutions

• Provide data-rich environment to evaluate system-level tradeoffs of sensors with different performance points

• Experiment with human-machine interaction in realistic simulated world environment

• Assess against varying terrain and threat contextsEnables closed-loop system tests prior to deployment to

reduce risk• Adapt high fidelity sensor simulations to Insight system• Merge recorded, live, and simulated data to enable realistic

system test and performance analysis

12

Virtual environment • Simulated sensor data• Simulated real world• Augmented collected data

Physical Test Bed

Coupling a virtual world to the real world


Insight Test BedComponent 3: Physical test bed

Populates the development incubator with data & is used to derive parameters for the virtual environment, from multiple data collections in a militarily relevant environment

Validates the development incubator and virtual environment results against validated scenarios executed in a militarily relevant environment

Used to develop and evaluate CONOPs and CONEMPs necessary for successful Service transition

Leverage physical test bed• Facilities to house, operate, exercise, and compute within

an operational infrastructure• Physical environment reflective of operational settings• Sources of clutter reflective of operational settings• Civilian and military personnel, and equipment sufficient to

execute validated scenarios• Infrastructure to capture all “truth” required for evaluation

Capability to electronically incorporate other remote facilities that provide unique capabilities via existing Secret/Defense Research Enterprise Network

13

Physical test bed• National Training Center• Realistic operational conditions

Defense Research Enterprise Network• Secure links to other facilities• Extended operational conditions


The National Training CenterFort Irwin, CA

14

Kirkush

Hamadan

Ahvaz

Dezful

H3

H1

H2

10

Arbil

An Najaf

Ramadi

Ar Rutbah

Karbala

Al QaimHadithah

Dam

Mosul

Tall Afar

Kuwait City

H3

H1

H2

10

Al Kut

Al Amarah

As Samawah

Al Basrah

Hadithah

Al Hillah

Fallujah

Dahuk

Az Zubayr

Al Qadisiyah

An Nasiriyah

Baghdad

Tikrit

Samarra

Kirkuk Sulaymaniyah

Bashur

Baqubah

SYRIAAl Hasakah

Dayr az Zawr

Al Mayadin

An Najaf

Al Muthanna

Al Basrah

Dhiqar

Maysan

Wasit

Babil

Karbala

Baghdad

Diyala

Al Anbar

Hadithah

Salah ad Din

At Ta’mimAs Sulaymaniyah

Ninawa

Arbil

Dahuk

SAUDIARABIA

IRAN

Ad Diwaniyah

MND-W

MND-NC

MND-S

MND-N

MND-B

5Source: NTC Command BriefFOR OFFICIAL USE ONLY - Distribution authorized to U.S. Government Agencies only

Realistic training environment used by ground forces to prepare for Iraq and Afghanistan rotations

Large training area located in the high Mojave Desert

US Army’s premiere Combat Training Center

Dedicated to training Brigade Combat Teams prior to deployment

Extensive existing instrumentation and infrastructure

Realistic OPFOR and clutter environment

Cooperative leadership permits experimentation without training interference

Major military exercises conducted monthly

Robust operational environment, tailored to both Iraq and Afghanistan

Scale

10km

EnvironmentBrigade: ~6000Officer Controllers: ~400Tactical Vehicles: ~2,000Role Players Population: ~3,000Civilian Vehicles: ~300Insurgents: ~300Civilian Main Post Population: ~10,000Main Post Military Population: ~ 3,000Commuting Workforce: ~2,000Area: ~300 sq kilometers


Insight BAA Technical Areas of Interest

15

1. Integration, Processing and Data Archive Environment

2. Adaptive Multisource Exploitation System

3. Collection and Resource Management System

4. Unified All-source ISR Human-Machine Interface

5. Physical Test Bed6. Virtual Environment7. **Development

Incubator8. **Transition Discovery

and Oversight9. Accelerated Innovation

** Technical Areas 7 & 8 will not be solicited under the Insight BAA


Notional Insight Program Organization

16

Virtual EnvironmentSingle Prime

Physical Test Bed& Test Integration

Single Prime

Unified All-source ISR HMI

Single Prime

Adaptive Multisource Exploitation System

Single Prime

Integration, Processing & Data

ArchiveSingle Prime

DARPAInsight Program

Manager

Collaborating Program Managers

Architecture Leadership Group

All Contractor Principle Investigators

Development IncubatorSingle Prime

Government or FFRDC

ContractingContracting Officers’

Technical Representatives

Transition Discovery& Oversight Team

Single PrimeGovernment or FFRDC

Senior Advisory GroupService Stakeholders

IC StakeholdersEmeritus Scientists

Science & Engineering Technical Advisors

Evaluation Leadership Group

All Contractor Evaluation Leads

Collection & Resource ManagementSingle Prime

Accelerated Innovation

Multiple ContractsCommercial & University

“Unity of Effort” via ACAs and contract incentives


FY10 FY11 FY12 FY13 FY14 FY15 FY16

Phased Development Plan

17

Field Test1 23 Oct through 02 Nov 2010

Field Test 2

Release Phase 1 BAAQ4FY10

Phase 1 leverages FT 1

Collaborations

Field Test 3‐9

24 month phase 1 30 month phase 2

TRANSTIONTRANSTION TRANSTION

TRANSTIONTRANSTION

PERSEASTAILWIND

FITTGuardDog

RC2

HARTWANDARGUSVADERVIRAT

Apr 2012

Mar 2011

Phase 1 Characteristics

Capabilities Demonstrations Exploitation System Test Bed

• HVI tracking with multiple conditional constraints

• Long term HVI tracking in heavy clutter

• Estimating & predicting the enemy network

• Ability to correlate across multiple sensor sources; WAMI, Radar, COMINT, OSINT, crowd-sourcing, text-based, and NFOV FMV sensors

• Incorporate automated resource management

• Provide user interaction though an advanced human-computer interface

• Unified workstation that integrates human and machine processing, including visualization, hypothesis manipulation, crowd-sourcing, and on-line learning

• Automated correlation, behavioral modeling, and threat network analysis across multiple sources

• Baseline control algorithms

• Assemble collected, simulated and processed data supporting tasks

• Assemble metrics & test harness• Evaluate collected, simulated and

processed data supporting new algorithms and CONOPs/ CONEMPs integration

• Evaluate correlation, control, human-machine algorithms on test + variants of each sensor

Release Phase 2 BAAQ4FY12


www.darpa.mil


InsightIndustry Day

Global ISR Data Collection #1

Dr. Michael PagelsProgram Manager

Information Innovation Office

National Rural Electric Cooperative Association Conference Center 21 & 22 September 2010


GISR-DC 1Sensor Systems & Processed Products to Insight

2

VADER (GMTI)

ARGUS-IS (WAMI)

SIGINTSIGINT

SIGINT

MX-14 / EO

SENSOR SYSTEM

PROCESSEDPRODUCT TO Insight

VADER • Target detections, locations & time tags• STANAG 4607 MTI & VEGS Tracks

ARGUS-IS • Full FOV Mosaic (in JPEG 2k format)

• Archival imagery: JP2K frames + headers

• Video window imagery: JP2K frames + headers

TAILWIND • Stitched LWIR imagery• Local area contrast enhancement

• De-noise• Temporal filter• Sharpening • Tracks• 3-D models• Alerts• Video summary

SIGINT Deanonymonized geolocations

SIGINT Geolocations

SIGINT •Device geolocations with IDs

•Device tracks

MX-14 / EO •Single FMV stream•EO imagery nadir to MX-14

•Surface Model•Orthos

JSTARS Tracks

JSTARS (GMTI)

TAILWIND (EO/IR)


GISR-DC 1Airborne Systems

3

Airborne Systems

Operating Altitude (MSL)

23000 ft

18000 ft

14000 ft

10000 ft

9000 ft

8000 ft

7000 ft

SIGINT

ARGUS

SIGINT

TAILWIND

SIGINT

Antennas at ground station shelters @

Building 6223

VADER

MX-14/EO

VADER flight path

(15 km offset)

Typical collection flight paths


GISR Data Collection #1Focus Areas/Scenarios

GISR-DC 1AOIs & Scenarios

4

Medina Wasl

Medina Jabal

Medina Irwin

Calibration Box

FOB DetroitCitadel

Scenario1Urban

Scenario 2Urban/Rural

Scenario3Small

urban/Rural

Scenario 4Small

urban/urban/Rural


GISR-DC 1Insurgent Scenarios

5


GISR-DC 1Scenario 1: Urban – Medina Irwin

6


GISR-DC 1Final Product Delivery

7

System Onsite Data Post? Final Product & Delivery Mechanism

VADER •STANAG 4607 data; Raw IQ data; “Quicklook” data•VEGS track & shape files

No Duplicate HDs delivered to VDL

ARGUS-IS Imagery organized by focal plane array (when archived, organized by time); Video streams sent to ground

No Duplicate HDs delivered to VDL

TAILWIND Imagery from 4 IR cameras; Imagery from 1 VNIR HSI; Imagery from 1 EO

Yes Stitched LWIR imageryDuplicate HDs delivered to VDL

SIGINT •Binary signal data broken in time sliced files•ASCII event metadata

Yes •Geolocations with error ellipses & IDs (temporary)•Temporary->permanent ID mappings•Event metadata merged with above data to report permanent IDs

•Labeled geolocations delivered to VDL

SIGINT RF data, tracking data, receiver metadata, tipping data

Yes Labeled geolocations delivered to VDL

SIGINT Ground Live geolocations; raw IQ data No Labeled geolocations delivered to VDL

SIGINT DF data & INS Yes Geolocations & tracks; updated geolocations with IDs delivered to VDL

MX-14 Single FMV stream (from 6 EO cameras) No Copied and delivered to VDL

EO imagery nadir to MX-14 DLTV & spotter video files

No Copied and delivered to VDL

Truth Vehicle track files & personnel/dismount track files; event logs

Yes Merged annotated tracks delivered to VDL

National SIGINT TBD No TBD

National IMINT TBD No TBD


GISR-DC 1Schedule

8

NTC Schedule

GISR DC 1 Collection Schedule

GISR DC 1 Schedule

DARPA Insight Industry Day Briefing

Documents

Transcript of DARPA Insight Industry Day Briefing