A Binary Agent Technology for COTS Software Integrity

Richard Schooler Anant Agarwal

InCert Software

The Mission Critical Environment

Output

Input OperatingSystem

SAP

COTSBinary

The developmentenvironment The deployment

environment

COTSBinary

Objective

COTSBinary

Input

Output

OperatingSystem

SAP

To improve the integrity of thedeployment environmentwith COTS softwarein the presence ofattacks, bugs

Our Focus

COTSBinary

Input

Output

OperatingSystem

SAP

Outer security defences will be breached by attackers

Use a practical, systems level approach – execution-time monitoring

On COTS program or data corruption, rapidly

d- detect problemsa- trigger an alarmp- try to protectr- recover

Our Approach: Execution-Time Monitoring of COTS through Binary Instrumentation

The developmentenvironment

The deploymentenvironment

COTSNewMissing

sourceLegacy

COTS Binary

COTSNewMissing

sourceLegacy

d- Policy specs for detectiond- Heartbeat insertiond- Argument range checksd- Rare code execution/sigs.a- Alarm messages to consolep- Defaults for fault tolerancep- Access constraints, redund.r- Logging

COTSNewMissing

sourceLegacy

Our Approach to Integrity A pragmatic system level approach Works with COTS binaries, even legacy codes Allows a user to establish desired security levels and

to some extent modify policy on the fly Allows monitoring of all user code execution, not

selected OS calls Can work completely at the user’s deployment site

Drawbacks of Binary Insertion Specific to a single platform, needs new technology

development for different platform Challenging to relate low-level observable events

back to high-level user actions hard to detect some types of intrusions that only

affect data corruption hard to protect or correct problems at higher

semantic levels

Three Major Components in the Prototype,Three Major Tasks

Core technology for customizable agent insertion into PC/NT, PC/Linux

Anomaly detection and reporting Rapid recovery and problem pinpointing

Selected Challenges and Ideas Core technology for agent insertion into binary

How to deal with unknown relocations, e.g., for dusty decks – incremental control and dataflow analysis; an integrated static and dynamic method

Dealing with multithreading, in particular, time syncing and monitoring events in a distributed environment

Anomaly detection Runtime comparison against execution path signatures? State machines for control flow checks (e.g., Abraham) How to minimize runtime overhead – borrow compiler optimization

techniques (e.g., steal registers, inline code, sampling, multilevel checks) Rapid recovery and problem pinpointing technology

Instrumentation records trace in circular buffer to pinpoint problem and facilitate recovery from crash/alert following an actual or suspected attack

Can we get data values? Use dataflow analysis and offline simulation to obtain intermediate data values

Measures of Success Core technology for agent insertion into binary:

Can we handle all binaries, even dusty decks? Target: Performance degradation to be under 1 percent

Anomaly detection What fraction of injected problems can we detect

automatically with user spec

Rapid recovery technology Performance degradation to be under 1 percent Can we cut recovery time significantly? We will measure

recovery time with and without As a bonus, can we catch problems before system goes down?

We are building a prototype system, work with real users, and measure

Current Progress Work on NT binary insertion prototype ongoing Demo of early capability showing

instrumentation simple recovery log detecting application has crashed taking control and writing out log

Handling multithreading, DLLs imminent Ongoing thinking on detection capability

AGENTtest al,0x3jnz 0x1143

AGENTadd ebx,ecxjc 0x1101

AGENTshr edx,0x1add ebx,edx

AGENTtest al,0x3jnz 0x1143

AGENTinc eaxadd ecx,ediadd edx,esicmp eax,0xa

1

2

3 4

5

... 1 2 4 5

while ((c = ++ci)) { INSTRUCTION_ITERATOR ii = c->Instructions(); while ((inst = ++ii)) inst->Lift(null_state); while ((inst = ++ii)) inst->Lift(null_state);

while ((c = ++ci)) { INSTRUCTION_ITERATOR ii = c->Instructions(); while ((inst = ++ii)) inst->Lift(null_state); while ((inst = ++ii)) inst->Lift(null_state);

Instrumentation for Recovery: Basic Idea At each program block, record the progress of program

execution.

Instrumentation Runtime

Naïve implementation:

•Write block id

•Increment index (in memory)

•Check for wrap-around

Too big/slow: many instructions per block; large buffer

DAG Header

•setup new trace record

DAG Block 1

•set bit in current record

DAG id

2

3 4

5

DAG id

DAG id

Instrumentation Example 1

test al,0x3jnz 0x1143add ebx,ecxjmp short 0x1156shr edx,0x1add ebx,edxinc eaxadd ecx,ediadd edx,esicmp eax,byte 0xajc 0x1101

Original

Instrumentation Example 2

mov eax,[0x300c]mov ecx,[byte eax+0xc]mov [byte eax+0xc],dword 0x1mov edx,[byte eax+0x8]mov [dword eax+edx+0xffe],ecxadd edx,0x4and edx,[byte eax+0x4]mov [byte eax+0x8],edxtest al,0x3jnz 0x1143or [byte ebp+0xe],byte 0x2add ebx,ecxjmp short 0x1156or [byte ebp+0xe],byte 0x1shr edx,0x1add ebx,edxor [byte ebp+0xe],byte 0x4inc eaxadd ecx,ediadd edx,esicmp eax,byte 0xajc 0x1101

Instrumentation

Original

Instrumentation Example 3

mov eax,[0x300c]mov ecx,[byte eax+0xc]mov [byte eax+0xc],dword 0x1mov edx,[byte eax+0x8]mov [dword eax+edx+0xffe],ecxadd edx,0x4and edx,[byte eax+0x4]mov [byte eax+0x8],edxtest al,0x3jnz 0x1147or [byte ebp+0xe],byte 0x2add ebx,ecxjmp short 0x1164or [byte ebp+0xe],byte 0x1shr edx,0x1add ebx,edxor [byte ecp+0xe],byte 0x4inc eaxadd ecx,ediadd edx,esicmp eax,byte 0xajc 0x1103

Instrumentation

Displacement Fixup

Original

Instrumentation Example 4

mov eax,[0x300c]mov ecx,[byte eax+0xc]mov [byte eax+0xc],dword 0x1mov edx,[byte eax+0x8]mov [dword eax+edx+0xffe],ecxadd edx,0x4and edx,[byte eax+0x4]mov [byte eax+0x8],edxtest al,0x3jnz 0x1147or [byte ebp+0xe],byte 0x2add ebx,ecxjmp short 0x1164or [byte ebp+0xe],byte 0x1shr edx,0x1add ebx,edxmov ecp,ebpor [byte ecp+0xe],byte 0x4inc eaxadd ecx,ediadd edx,esicmp eax,byte 0xajc 0x1103

Instrumentation

Displacement Fixup

Original

Register Transfer

Related Approaches Applied at source level during development

e.g., type based safety; work of Lee et al. Applied at link time with special object formats

e.g., software fault isolation; work of Pandey et al. Applied through interpretor

e.g., safe Java interpreters Applied during program execution – works with existing COTS

e.g., sandboxing through middleware software e.g., modify OS with extensions to catch OS calls,

Ghosh e.g., wrap application to steer OS calls to

intermediary, Ghosh

The Current Commercial Solution: Middleware

COTSBinary

Input

Output

OperatingSystem

SAP

Middlewarewrappers

•Slow•Cannot deal with viruses•Cannot improve faulttolerance of COTS package itself

Sandboxing through OS Call Interception(OS extensions or wrapping executable)

COTSBinary

Input

Output

OperatingSystem

SAP

•This is the right first step, and less intrusive than binary instrumentation

•Cannot improve fault tolerance of COTS package itself

•Cannot handle communication through shared memory

•Cannot observe internal app behavior

•OS extensions can slow OS for others

Summary A systems approach to COTS Integrity Approach based on execution-time

monitoring using binary insertion We have an early prototype version of NT

binary insertion implemented