1

Collision of events…

2

Typical Network Incident Response

Technicians must choose: Expend effort collecting forensically sound data, or Simply restore network as quickly as possible

Evidentiary files altered in the process Forensic value limited

Expediency wins…and so do attackers!

3

New Zealand vs. Russian Cases

Characteristics NZ Hacker Case Russian Hacker Case

Type of attack Typical script kiddie

intrusion scenario Online criminal automated

auction scam

Damages $400,000 $25 million

Investigator time 417 hours 9 months

Consequences Community service 3 & 4 years in Federal prison

4

Lack of interest in prosecution

Inordinate effort/cost of investigations

Poor legal outcomes

Investigations not scalable Too expensive Too labor intensive Ties up brilliant technical minds Little comes of it

5

Growing Threat Spectrum

6

The Escalation Tendency of the Hacker Arms Race

7

Fueling the "arms race"

The volume of cyber attacks continues to increase.

It takes less technical knowledge to launch increasingly sophisticated attacks, using increasingly sophisticated hacker tools.

Organizations are becoming increasingly reliant on public networks, often without tempering enthusiasm with a concern for security

Surveys continue to report increased organizational investments in tools and techniques that protect information systems and prevent intrusions in response, yet criminal intrusions are escalating in number and severity.

8

Expect the appetite for prosecution to change

$$$$$$$$$$$$$$$$

9

The Problem

Why this problem must be solved

10

Frye / Daubert Standards

Frye Standards: Is the approach sufficiently

established?

Has the technique gained general acceptance in its field?

Does it require study/experience to gain special knowledge?

Does expertise lie in common experience/knowledge?

Daubert/Kumho Factors: Has the technique used to collect

evidence been tested? (or, can it be tested?)

Has the theory underlying the procedure, or the technique itself been subjected to peer review and publication?

Does the scientific technique have a known or potential rate of error?

Do standards exist, along with maintenance standards, for controlling the technique’s operation?

11

Expert Witness Testimony

The challenge:– Collect/store forensic data– Present forensic data credibly in court

Admissibility standards Frye v. United States. 293 F. 1013 (D.C. Cir. 1923) Daubert v. Merrell Dow Pharmaceuticals, Inc. Daubert, 509 U.S. 579 (1993) (further enunciated in Kumho Tire Co. v. Carmichael) Rule 702 (Federal Rules of Evidence)

12

Foundation

Expert believability based on jury trust

Experts either– Explain evidence so a jury can understand or– It’s so complex, only an expert can understand

Opposing counsel discredits witness by challenging testimony's foundation—

– 'how do you know this?’; – 'how can you say this?'; – 'how can we believe the validity of what you say?‘

Radar gun analogyThe Genuine Tipmra Speeding Ticket Defensehttp://www.tipmra.com/new_tipmra/washington_state_speeding_ticket.htm

13

Computer Forensic Tool Testing Project (CFTT-NIST)

"…to establish a methodology for testing computer forensic tools by the development of functional specifications, test procedures, test criteria, test sets and test hardware.“

Scope: 'software and hardware tools used by law enforcement agencies to acquire data from digital storage media'

Gap: Network devices that collect/gather data

14

Problem

…the courts may begin to expect the same high standards to which they've become accustomed for the preservation of evidence on computer hard drives, when evidence is gathered on complex networks or captured in transmission.

(Sommer, September 2002)

15

Rationale

Experts must speak competently about forensic data reliability– Skills of data gatherer – Process used– Devices employed

Establishing soundness of network data gathering devices can– Support prosecution/defense– Assist pursuit of legal remedies

BUT manufacturers rarely provide conclusive information– Proprietary design– Expense of calibration– As yet no demand

FURTHER manufacturers specifications are not reliable

We expect this to change…..

16

Consequences

A justice system subject to confusion—as innocent individuals are wrongly convicted and those deserving of punishment get away with criminal acts,

Escalating growth in online crime—as prosecution cases fail due to inadmissible evidence and digital crimes go unpunished,

Growing liability for companies—as sensitive customer information and digital assets are vulnerable to increasing online theft and as internal misusers challenge employee disciplinary action supported by

questionable digital evidence,

Decreasing trust in the e-economy—as companies and customers reassess doing business over public networks, and

A general halt to the progress of the Information Age—as online business and communications are no longer viable [FH07].

17

In the meantime…

No standards

No testing labs

Unreliable specifications

Network evidence admitted anyway

First responders still responsible

18

Proposed Solution

Develop device calibration standards

Comparison of instrument performance to a standard of known accuracy in order to determine deviation from nominal and/or make adjustments to minimize error

Start with user verification tests– Use current network testing protocols– Establish calibration approach

19

Calibration

"I often say that when you can measure what you are speaking about and express it in numbers you know something about it; but when you cannot express it in numbers your knowledge is a meager and unsatisfactory kind; it may be the beginning of knowledge but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be."

Lord Kelvin lecture to the Institution of Civil Engineers 3 May 1883[1] [4]

[1] Lord (William Thomson) Kelvin--scientist, engineer and pioneering metrologist--is associated with the development of the Kelvin temperature measurement scale

20

The Problem

"…the courts may begin to expect the same high standards to which they've become accustomed for the preservation of evidence on computer hard drives, when evidence is gathered on complex networks or captured in transmission." [Som02]

– Computer (disk) forensics – more developed science Disks seized by law enforcement Investigators trained in legal procedures Tools, procedures Data accepted in court

– Network forensics – can’t “bag and tag!” Crime scene a live network “Investigators” often untrained network administrators Tools developed for other purposes – troubleshooting, tuning, etc. Data admitted anyway

– Sophistication on both sides of the bar is growing – expect challenges!

21

Consequences

A justice system subject to confusion—as innocent individuals are wrongly convicted and those deserving of punishment get away with criminal acts,

Escalating growth in online crime—as prosecution cases fail due to inadmissible evidence and digital crimes go unpunished,

Growing liability for companies—as sensitive customer information and digital assets are vulnerable to increasing online theft and as internal misusers challenge employee disciplinary action supported by

questionable digital evidence,

Decreasing trust in the e-economy—as companies and customers reassess doing business over public networks, and

A general halt to the progress of the Information Age—as online business and communications are no longer viable [FH07].

22

Rationale for Calibration Focus

Without calibration of network devices used to collect forensic data, the data is:

Subject to serious legal challenge and At risk for inadmissibility in court proceedings [ECF07, Som02].

Calibration not currently performed: Proprietary architecture and forwarding algorithms Troubleshooting, network tuning functionality focus Collecting admissible evidence not primary No standards for device validation

23

Computer Forensic Tool Testing Project (CFTT-NIST) Established

Established in anticipation of legal challenge

Mission to develop testing methods to evaluate computer forensic tools

Scope limited to 'software and hardware tools used by law enforcement agencies to acquire data from digital storage media'

Gap: Enterprise network devices used collect forensic data out of scope

24

Rationale for Developing Network Device Calibration Methodology

Need to establish reliability of network data gathering devices

Need to provide conclusive information that manufacturers don’t provide FURTHER manufacturer specifications are not reliable

Courtroom challenges to network devices used to collect evidence is expected

Yet, no calibration standards/third party labs exist

Network evidence admitted anyway

First responders still responsible

25

Proposed Solution

Develop network device calibration standards

Start with user verification tests– Use current network testing protocols– Establish calibration approach

26

Summary of Progress

27