M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

1

A Guide to Measuring the ROI of Security Orchestration and Automation

O V E R V I E W

Today’s organizations expend a significant amount of time and money on technologies that will help them reduce

risk. Despite their efforts, measuring the efficacy of security tools and the incident response process remains a

challenge. The goal of this paper is to demonstrate how automating incident response can help your SOC in three

key areas: people, process, and technology. The paper will discuss key metrics that you can use today to

understand how you’re performing right now so you can begin to quantify improvement.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

2

Table of Contents

A Guide to Measuring the ROI of Security Orchestration and Automation ......................................... 1 Overview ...................................................................................................................................................1

Cybersecurity is Evolving .................................................................................................................. 3

Stage One: Prevention ..........................................................................................................................................................3

Stage Two: Detection ............................................................................................................................................................3

Stage Three: Response ..........................................................................................................................................................4

Understanding the Challenges Organizations Face ......................................................................................4

The Capacity Challenge .........................................................................................................................................................4

Time Can Be the Enemy ........................................................................................................................................................5

What Do You Measure?.................................................................................................................... 6 People .......................................................................................................................................................6 Technology ................................................................................................................................................7 Process ......................................................................................................................................................7

Calculating Return on Investment ..................................................................................................... 8 Number of alerts ........................................................................................................................................8 Analysts and Costs .....................................................................................................................................8 Analyst Capacity ........................................................................................................................................9

Calculations ...........................................................................................................................................................................9

Total Analyst Cost ......................................................................................................................................9 Total Alerts Investigated ............................................................................................................................9 Percentage of Alerts Investigated ...............................................................................................................9 Analysts Needed for 100% Coverage ...........................................................................................................9 Current Cost Per Investigated Alert........................................................................................................... 10 Cost Per Investigated Alert – 100% Coverage ............................................................................................ 10 Automation Cost ...................................................................................................................................... 10 Cost for 100% Coverage............................................................................................................................ 11 Cost Per Investigated Alert ....................................................................................................................... 11

Conclusion ..................................................................................................................................... 11

About Hexadite .............................................................................................................................. 12

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

3

Cybersecurity is Evolving The way we approach cybersecurity is evolving. The changing nature of cyber threats and the increase in volume

has led to new processes and technologies. Let’s take a walk through each stage to see how cybersecurity has

evolved.

Stage One: Prevention

As recent as 10-15 years ago, we focused mainly on prevention. It was all about trying to keep the bad guys out of

our networks. If we’re being nostalgic about this time period, one might describe the prevention stage as blissful.

The volume of attacks was far lower than they are today, allowing small teams to put all of their focus on

prevention. Even if something got through-maybe an email virus-the disruption might be a few hours at the very

worst. SOCs had the resources to simultaneously focus on prevention and then deal with threats on an as-needed

basis.

Stage Two: Detection

As bad actors and attacks became more persistent and the frequency, complexity, and volume of attacks increase,

detection became imperative in order to know when something got past our defenses. SOCs began to evolve into

something much more robust—instead of focusing on just prevention, we began using an effective combination

of antivirus, SIEM solutions, and other detection systems to identify potential malicious activity.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

4

Stage Three: Response

Today, we assume compromise. And while the prevention and detection approaches of the past have relied

largely upon automated systems and technology, response depends largely on people and manual processes.

Throughout this evolution, the volume and complexity of the attacks and the subsequent alerts have increased

exponentially. Where we could get by in the prevention era with a small team, we’re now overwhelmed by the

number of alerts we see to the point where we can only investigate a very small percentage of what we see. The

drive toward automation is primarily attributable to a capacity mismatch.

U N D E R S T A N D I N G T H E C H A L L E N G E S O R G A N I Z A T I O N S F A C E

Today’s security operations and incident response teams are locked in an unfair fight. Resources are short, time is

tight and prioritization—the only way to manage the ever-increasing volume of alerts—is a major security risk.

Automated cyber-attacks are driving this disparity. And yet, most cyber-analysts are still using manual tools and

writing custom code to mitigate threats.

The Capacity Challenge

Security teams simply don’t have the capacity to match adversaries. Research from analyst firm EMA found that

92% of companies receive 500 cyber alerts or more each month. That’s equal to 15,000 per month. If one cyber analyst

can investigate roughly 10 alerts per day – one at a time – that’s 300 alerts investigated per month.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

5

If you wanted to investigate every single alert, here’s what you’d need: 50 cyber analysts working 8 hour shifts 7

days a week just to keep up with current alert volume. That’s not realistic.

Due to a well-documented worldwide cybersecurity skills shortage, there are 1 million vacant cybersecurity jobs

globally. Companies simply cannot hire the capacity problem away.

Time Can Be the Enemy

Because of the mismatch between alerts and capacity, attacks often are ignored. 80% of data breach victims don’t

realize they’ve been attacked for a week or longer, with attacks often going more than 200 days before being

discovered. That’s a lifetime for attackers, and there are numerous high profile examples that show the damage

that can be done.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

6

What Do You Measure? To determine your current performance, we’ll look at three key areas, and within each area we’ll explore metrics

around performance, risk, and how to quantify improvement.

P E O P L E

How do you measure your team’s performance when it comes to security?

Performance

The first thing to measure is the number of alerts they investigate to

understand capacity. We’ve seen that the average cyber analyst can handle

anywhere between 8 and 12 investigations per 8 hour day. Of course, that

number varies based on the kind of alert, but that’s the number we’ve heard

from customers.

Risk

When it comes to risk, you’re really trying to understand the cost and

replacement cost of your analysts. A recent report states that 50% of

cybersecurity professionals are contacted by recruiters more than once a week. The shortage is real, and it’s very

difficult to retain people when they’re not being challenged and are doing repetitive work. When you look at the

cost associated with recruiting, training, and the time it takes to get a new analyst up and running, it’s easy to see

how valuable it is to keep good analysts. Out of the three categories, people is the one category where you can

have both the greatest impact and return.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

7

T E C H N O L O G Y

Performance

Look at all of your detection systems to establish a baseline of how many alerts are

coming in as well as the percentage of those alerts that are false positives.

Risk

When security teams lack the capacity to respond to all alerts, they often tune

down their detction systems to match what they can handle, dramatically

increasing the odds that attacks will go unnoticed. What thresholds have you set to

prioritize the alerts you’re receiving? When it comes to prioritization, you’re really

just setting a score by which anything that doesn’t hit that measure gets ignored.

Understanding how much of what your technology is detecting gets ignored will help you understand how

automation can improve your security using an objective measure.

P R O C E S S

Performance

The big three measures for judging an incident response process are:

Mean time to notify: How long does it take from a threat being detected to

the time someone is notified? What’s important to note here is that idea that

it isn’t the time it takes to fire off an alert, but instead, the time it takes for

an analyst to get to that alert. One of the main issues is that people can only

really single-task. If your analysts are already working on an investigation,

they’re at full capacity, and they may not get to a new alert for hours at

best. How long does that alert sit in the inbox waiting?

Mean time to investigate: Once alerted, how long does it take to start the

investigation, and what’s the average investigation duration?

Mean time to remediate: If an investigation finds something requiring action, how long does that take? A

good example is a Trojan. Once detected, how long does it take your analyst to hit the endpoint,

determine the processes running, look at any IP addresses the endpoint is communicating with, delete the

files, kill processes, add a firewall rule, investigate other impacted machines, etc.?

Risk

What percentage of alerts are never investigated? That’s your baseline number to assess your risk and to

understand how you can improve.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

8

Calculating Return on Investment Before Automation

Let’s walk through an example of how a company can measure their current incident response performance and

the ROI they can expect from an automated solution.

Inputs

N U M B E R O F A L E R T S

The first input we’ll need is the alert volume, and this number varies based on several factors. First, some

detection systems will send an alert for each potential incident found while others will send one alert for multiple

related events. The number we reference most often is from an EMA report that states 92% of enterprises see 500

or more alerts per day.

A N A L Y S T S A N D C O S T S

Next, we’ll need to understand the number of FTEs that are dedicated to investigate alerts and their fully loaded

cost. While the Bureau of Labor Statistics lists the median cyber analyst salary at just over $90,000 in 2015, our

customers report a much higher number when factoring in items like:

Recruiting Cost: Based on the worldwide cybersecurity skills shortage, most companies retain the

services of a recruiter to fill cybersecurity roles, adding a minimum overhead of 15% to the salary cost

Training Cost: The additional expenses related to training analysts on the company’s systems and

processes

Bonuses: Retention and goals-based bonuses

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

9

A N A L Y S T C A P A C I T Y

How many alerts can your analysts investigate in a given 8-hour day? We’ve seen a range of 8-12, again

depending on how alerts are defined. While some alerts are low fidelity and can take minutes to dismiss, others

take many hours to investigate, incriminate, and remediate.

Calculations

T O T A L A N A L Y S T C O S T

A simple calculation—we just multiply the cost/analyst by the number of analysts. In our example scenario, we

have 5 analysts at a fully loaded cost of $125,000 each per year, bringing our total analyst cost to $625,000 per

year.

T O T A L A L E R T S I N V E S T I G A T E D

In the example scenario, we see that we have 5 full time analysts, each capable of investigating 10 alerts per day,

bringing us to a total of 50 alerts investigated daily.

P E R C E N T A G E O F A L E R T S I N V E S T I G A T E D

With 500 alerts per day and the capacity to only investigate 50, our example company is only investigating 10% of

their total daily alerts, ignoring a full 90% of what their detection systems are finding. The company pays

$625,000 to investigate just 10%.

A N A L Y S T S N E E D E D F O R 1 0 0 % C O V E R A G E

Using the standard 10 alerts per day, the company would need 50 analysts to handle the alert volume, and at

$125,000 per analyst per year, it would cost the company more than $6.2 million per year to cover 100% of alerts.

And if 60% to 80% of those alerts are benign, spending over $6 million to investigate them all would be

unfathomable.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

10

C U R R E N T C O S T P E R I N V E S T I G A T E D A L E R T

By dividing the total annual analyst cost ($625,000) by the total number of alerts they can investigate (15,000

annually), we see that the company currently pays $41.67 for each investigated alert.

C O S T P E R I N V E S T I G A T E D A L E R T – 1 0 0 % C O V E R A G E

If the company were to hire enough analysts to investigate 100% of alerts, the cost for all 50 analysts ($6.2 million

per year) to investigate all alerts would equal $34.25 per investigated alert.

After Automation

Adding automation assumes the ability to investigate a full 100% of alerts from detection systems. Let’s look at

the ROI.

A U T O M A T I O N C O S T

The cost of an automation solution can vary wildly from vendor to vendor. Some charge per set, per investigation

and others charge based on the number of endpoints covered.

In the example above, we calculated the cost of automation by using 10% of the cost it would take to hire enough

analysts to cover 100% of the alerts they see.

This is merely illustrative and not intended to represent the pricing structure of Hexadite or any other vendor.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

11

C O S T F O R 1 0 0 % C O V E R A G E

You’ll remember that in the previous example, hiring to capacity would have cost $6.2 million. Assuming the

$625 ,000 calculation and the yearly analyst cost of $625,000, the company would pay $1,250,000 to investigate all

alerts using automation.

C O S T P E R I N V E S T I G A T E D A L E R T

While the cost per investigated alert was $34.25 if the company hired the 50 analysts necessary to investigate

everything, using automation the cost per investigated alert drops to $6.85, a savings of $27.40 per alert

investigated.

Conclusion Using the inputs and calculations above, you will be able to clearly see whether the ROI of automating incident

response is worth the effort. To learn more about how Hexadite AIRs can help increase your investigation and

remediation capacity, security posture and drive down costs, visit Hexadite.com or click below to schedule a

demo.

M E A S U R I N G T H E R O I O F S E C U R I T Y O R C H E S T R A T I O N A N D A U T O M A T I O N

Intelligent Security Orchestration and Automation

12

About Hexadite Hexadite is the first agentless intelligent security orchestration and automation platform for Global 2000

companies. By easily integrating with customers’ existing security technologies and harnessing artificial

intelligence that automatically investigates every cyber alert and drives remediation actions, Hexadite enables

security teams to amplify their ability to mitigate cyber threats in real-time. For more information, follow

@Hexadite on Twitter or visit www.hexadite.com.