Mitigating the Insider Threat using High- dimensional Search and Modeling Telcordia Contact: Eric...

Mitigating the Insider Threat using High-dimensional Search and Modeling

Telcordia Contact:Eric van den Berg(732) 699 [email protected]

An SAIC Company

SRS PI Meeting, Arlington, VA, January 27, 2005

Team:Shambhu Upadhyaya, Hung Ngo (SUNY Buffalo)Muthu Muthukrishnan, Raj Rajagopalan (Rutgers)

SRS PI Meeting – 2

Talk Outline

Project Overview– Goal

– What is done now?

– Technical Approach

– Technical Challenges

– Quality Metrics

– Expected Achievements

– Task schedule and milestones

Project Progress– Design

– Lightweight Demo


Project overview

Project goal: to build a system that defends critical services and resources against insiders, which– Can detect attacks by correlating large numbers of sensor

measurements– Can synthesize appropriate pro-active responses to protect critical

services while minimizing collateral damage. What is done today?

– Reactive systems: Detect attacks late in cycle– Anomaly detection systems: Few streams for correlation, suffer from

curse of dimensionality– Human-in-the-loop systems: Response not scalable, prior attacks

pulled from administrator experience– Consequences of response v.s. impact attack: collateral damage

may be large


Project overview (continued)

Technical Approach– Large network of sensors, to let insider trigger alerts– High dimensional network state description in terms of sensor alerts– Search engine finds top-K historical states similar to sensor snapshot– Insider modeler and analyzer tool used to identify attack points, train search

engine, guide sensor placement– Response engine to analyze impact of potential attack on critical services

and synthesize reconfiguration response

Technical Challenges– Extensive experience with SVD based searches in text-based information

retrieval. Here we are testing search technology in a new domain– New ‘Insider analyzer’ key-challenge graph problem is hard– Training search engine, labeling and annotating states


Project overview (continued)

Quantitative Metrics to measure success and overheads– Keep track of detection performance: detection/false alarm rate

– Test detection for novel attacks which are variations of known attacks

Expected Major Achievements– New high-dimensional anomaly / intrusion detection system, which

can scale to large, internet-size networks

Task schedule and milestones

Task(milestone) Jan-Jun 05

Jul-Dec 04

Design (document) Prototyping (software) Testing (report)

Jul-Dec 05


Proposed architecture

Sensor

Sensor

Sensor

Sensor

Normalizer

Filter

Aggregator

SearchEngine

ResponseEngine

Network StateRepository

HostScans

AuditScans

StateData

TrafficMeasure-

ments

Reconfiguration

Top KList

RefinedQueriesN

ETWORK

High-DimensionalSearch

Insider Modeler

andAnalyzer

Organizationaldata

Labels andfilters for states

Post-processing


Sensor network design

Monitor critical services and applications, hosts, devices on which these depend:– Network sensors: aggregate traffic, network flows, histogram-

based, ‘sketch’-based.– Host sensors: applications, cpu-load, audit logs, web logs, user

challenges, profile anomalies, file-integrity checkers Design goals: scalability and extensibility

– Incorporate available sensors as well as newly developed sensors easily

Use data-aggregation and filtering to reduce data volume– Only store sensor alerts

Store sensor alerts in unified format– IDMEF-like database schema to store sensor data


Network state description

Network state is constructed from sensor alerts:– Accommodate heterogeneous sensor types

– Account for different sensitivity of sensor types

– Tolerate possibly delayed or missing, ‘out of order’ alerts

Alerts are mapped to a high-dimensional vector for search– Coordinates correspond to different sensor-alert types

– Some possibilities for mapping values: Total number of sensor alerts of given type in (sliding) time window Indicator: sensor alert occurred in (sliding) time window

Network state is labeled:– With Classification e.g. ‘Normal’, ‘DoS’, ‘Insider’

– With Response for Response Engine


Search engine design

Goal: Find historical documented network states most similar to the current network state snapshot

Output: Top-K list of ranked/prioritized similar states Ranking can be based on similarity metric and/or potential

impact, e.g. attack ‘risk’. Impact of historical network states is documented, impact of

current state can be analyzed/verified with the Response engine

Search engine reduces dimensionality of search space– Using Singular Value Decomposition, or random projection

Similar states found by nearest neighbor search using distance metric (e.g. cosine similarity, Euclidean distance)


Normal

InsiderDoS

Worms

High dimensional Search on Labeled States

1.0

S81.0

S13

S14

S4

S10

S15

S9

query

S1S2

S6

S16

S5S11

S12

S7S17

S3


Precursors: early detection of insider attacks

All attacks, including Insider attacks, need to be detected as early in the cycle as possible to minimize damage

Nearest neighbor search and/or cluster analysis may help label and diagnose vector-based network states

– but how to represent time evolution? Like learning attacks from documented historical network states, we can

also document attack precursors or attack stages – Full attack now represented as a sequence of network state vectors– Initially, we can obtain attack ‘cases’ from forensic analysis– Robust against slow attacks: no explicit dependence on time– Would like to make ‘precursor’ annotation (semi-) automatic

Two possible approaches to automatic precursor annotation– Temporal precursors: use (e.g. exponential) temporal decay functions leading

up to attacks to indicate confidence in network state as precursor– Spatial precursors: consider all state vectors occurring within a time window of

length T of an attack vector to be ‘correlated’, I.e., precursors.


Schematic for using precursors


Impact Analysis using Response Engine

Building upon Smart Firewalls technology from Dynamic Coalitions program

– Response Engine has overview of current network configuration– Response Engine logically validates Policies, expressed in terms of end-to-

end service availability– Response Engine generates candidate reconfigurations to comply with

Policies as much as possible In this project

– Detected attack type and location is translated into its effect on the stated policies and current network configuration

– E.g. Server failure due to a Denial of Service attack Response Engine can analyze the impact of both the attack and its

candidate responses on the availability of critical resources– E.g. Analyze impact of vulnerability exploit: how widespread is the

vulnerability? Administrator can push response into the network


Response engine design


Insider analyzer and modeler

Insider threat manifests in two forms:

– Insider abuse while staying within legitimate privileges– Insider abuse while exceeding assigned privileges

Focus on an insider's view of an organization such as hosts, reachability and access control

A new threat model called a “key challenge graph”

– Similar to attack graphs, but far less emphasis on details– Allows static analysis of insider threat on problem instances of manageable

size

Threat analysis metricCost of AttackActual targetTarget Vertex

Location of insiderStarting VertexAccess ControlKey Challenge

Information, CapabilityKey

Connectivity, ReachabilityEdgeHosts, PeopleVertex

AbstractionModel Component


Features of key challenge graph

Ease of representation– Unlike attack graphs, KCG mirrors actual network topology– Abstract info. in the form of hosts, people, channels, access control etc.– (May lose some accuracy, but it is a time-tradeoff)

Proactive scheme – Threat assessment– Works as a precursor to our online detection subsystem– Can also be used independently

Answers important questions such as:– Is the current security set-up sufficient?– If not, what are likely attack paths? – Additional security recommendations? What parts of organization need

additional monitoring? Which security policies need to be revised? Role and impact:

– Guide the placement of sensors for data collection– Cluster weighting and size reduction


Modeler and Auditor Program for Insider Threat (MAPIT) Tool Development


Screenshot


Detection of an insider attack via honeytokens

Insider Fin-data

CFOAdmin

Login: cfo @ fin-data

Password: ”makemoney”





ALARM!

Sensor 1 (NIDS)

Sensor 2 (HIDS)


Alert-id 1 Sensor-id 1 “Honeytoken access”

NORMAL

Alert-id 3 Sensor-id 2 “Honeytoken used”



Search engine detection

Network State Repository:

prior network statesand

responsesALARM!

Alert-id 2 Sensor-id 2 “Honeytoken used”


ATTACK

Search Engine

Mitigating the Insider Threat using High- dimensional Search and Modeling Telcordia Contact: Eric...

Documents

Transcript of Mitigating the Insider Threat using High- dimensional Search and Modeling Telcordia Contact: Eric...