Information Security Management Metrics Development

Sven 0/ofSandstrtm Herrera

Applus+ SIT

1 ABSTRACT

It is commonly accepted that you cannot improve whatyou can not measure.

This concept is applicable to almost all the departments ofan organization (financial, production, human resources,quality, etc.). However, in a lot of organizations theinformation system area, and more concretely theinformation security area, does not base its decisions onquantifiable, measurable, comparable and contrastabledata, but on the experience of its managers.

It is essential for an organization to place indicators forobtaining information on the information securitycontrol's effectiveness, so that problems can be detectedas soon as possible.

2 INTRODUCTION

Information Security Management Systems (ISMS) arebecoming more and more popular for the organizationsthat wants to improve their information security levels.

The ISMS is powered by a set of controls objectives andcontrols which are directly and proportionally related withthe results of a risk assessment process undertaken in theorganization. The effectiveness of the hole ISMS isdirectly conditioned by the effectiveness of theimplemented information security controls.

To have updated information available about theeffectiveness of the controls it is needed to obtainmeasures that allows the development of informationsecurity indicators, easy to understand and to update.

These indicators are supposed to provide the managementwith information that let them take decisions and adoptpreventive actions before incidents takes place.

3 MEASUREMENT AND MANAGEMENT

Conversations, such as the following are very frequent ina management environment, in fact essential:

.-How have we done this quarter?-Not so good. We have incremented sales by 90ocompared to the same period last year, butunfortunately our benefits are not grow ing,basically because we have increased the expensefor personnel by 5,5%.

If an organization does not have information about howits business is runing, it hardly will be able to becompetitive.

It is not only important to have information on the currentsituationl; it is also very important to be able to rank thesituatiol in comparison with that of last week, or last year,to identify whether improvements have taken place.

Probably the most amazing exception to this way ofmanaging operations based in data can be found ininformation security. If somebody asks in the informationsecurity department "Hi, How is it going on today?" inmany organizations they will not be able to say anythingmore than "Well, it runs, so ..., not bad".

4 MANAGED INFORMATION SECURITY

The information security market is embarking on themanagement model proposed by ISO 17799, UNE 71502,BS 77 99-2 and coming ISO 27001, and has discovered inthis model a successful way forward to improve theinformation security. An important element in thesestandards is to consider information security not only as arequirement of the data processing or information systemsdepartment; but to view it as a requirement relevant to allparts of the organization, in a similar way as QualityManagement Systems does.

From this perspective, all areas within an organizationmust be involved in the information security processes,and this requires a management oriented model.

Hence management, among other things, needsquantitative data to measure the effectiveness of theirengangement: how many employees, clients, projects,reclamations, taxes, defective pieces, etc did it take? The

0-7803-9245-O/05/$20.00 C2005 IEEE

key questions are "How much does all this cost?" and"What is the benefit achieved?"

According to this model of information securitymanagement an organization should establish amanagement framework which could include thefollowing elements:

1. Policies: As summary of the managementframework.

2. Standards: Management criteria that mustensure to support and enforce the policy.

3. Procedures: Procedures adopted to comply withthe standards.

4. Work and instruction orders: The specificdetailed tasks, specifying what, who, when andwhere carries out actions regarding theimplementation of security processes andprocedures.

5. Records: Evidence that the work orders andinstructions that have been developed to complywith the procedures, standards and policies are inplace and being used

Within this model, there are several ways for anorganization to detect security problems and deviations.This can found when reviewing and updating the riskassessment results, during a revision of the ISMS, at theISMS audits, or from the feedback received from theincident management process (obviously incidents cangive clear indications of where things are going wrong).

5 METRICS AND INDICATORS

Metrics and indicators intxoduce in the ISMS a moreproactive element:

1. Metries: Elements that provide quantitative dataon different aspects that can be useful to evaluatethe effectiveness of a security control.

2. Indicators: Combinations of the data providedby the metrics, so that they provides usefulinformation to the organization.

At this point there are frequent discussions aboutterminology, what it is important is not how it is call, butthe concept. Hereby metrics will be considered as thedirectly measured data, and indicators as the data orinformation derived from the metrics.

An important fact is that the organization gathers data thatcan be processed and converted into useful informationfor the organization's stakeholders, specially for themanagement to take decisions.

If, for example, an organization establishes an indicatorshowing the relation among the failed and correct loginstrials, and in the last week this indicator has increased by40 %, it can be due to the fact that:

1 There are new users and they have not yetfamiliarized themselves with the system.

2. The organization has changed its passwordpolicy for a more restrictive one and the users arestill becoming adapted to it.

3. Somebody is attacking system, for exampletrying to break a password with dictionry orbrute force attack.

Once the indicator has signalled that something abnormalis happening, the organization has the chance to reactbefore an incident takes place.

This is the true advantage of deploying these indicators,the organization can make decisions without a need towait until:

1. An incident is reported bya. Employeesb. Customersc. Providersd. Regulatory bodies

2. Risk assessment revision3. Incident register revision4. ISMS management review5. ISMS audits

It is also especially interesting to compare the actualsituation with previous situations. It might not be veryuseful to know that the security level of the organization(regarding the established criteria) is 7 out of 10, but iflast week it was 8, then the situation is getting worse.

5.1 Metrics and indicators types

From the management perspective the organization shouldplace information security management indicators thatprovide different categories of information according tothe object or area to which they will apply.

Each organization may have different criteria to establishthe most relevant categories of metrics and indicators, butthe following categorization can be considered for most ofthe organizations:

1. General: metrics and indicators not necessarilyrelated with information security, but that can bycombined with other information security metrics

to provide useful information, e.g. number ofPCs or employees in the organization.

2. Management: metrics and indicators concerning

the management of security controls, e.g. numberof employees with information securityresponsibilities.

3. Operation: metrics and indicators concerning

how the organization carries out those operationsin which users intervene, e.g. number ofinformation security incidents reported byemployees.

4. Technological: metrics and indicatorsconcerning how the organization carries outthose operations in which no users intervene, e.g.

average network broadband available.5. Environmental: metrics and indicators

concerning the situation of the organization'senvironment that may condition any of thesecurity characteristics of its information assets,e.g. total number vinuses in the wild in the lastweek.

6. Personnel training: metrics and indicatorsconcerning the training levels of the personneloperating in the organization, e.g. number ofCISA in the information security auditdepartment.

5.2 Selecting metrics and indicators

Information security metrics and indicators are selected in

order to have quantitative or qualitative data in an

understandable and useful form for their stakeholders.

In that sense, the organizations should identify enoughmetrics and indicators to be able to get to conclusionsabout the effectiveness of its information securitymanagement. Obviously the number of metrics andindicators to use will be proportional to the nature andscale of the organization's operations.

It is not so relevant if the information given throughmetrics and indicators is expressed as direct or relativemeasures or as indexed or added information, what it is

important is that provides information that managementcan understand.

For example, the number of computers with an antivirusinstalled, lets say 35 computers, may not be very useful,but the percentage of computers without antivirus gives a

more comprehensive information that a non technicalaudience understands. If the organization has 70computers, that means that 50% of their computers are notprotected which it is not acceptable, but if it has 1000

computers, that means 3,5% of unprotected computers,which may be acceptable.

The first thing that the organization should have veryclear is what is to be measured and what is its relativeimportance, e.g. Management commitment withinformation security.

When this is clear, the organization should select themetrics and indicators to be used to obtain the data and toget to information about what is going to be measured.

This step use to be quite complicated, What can Imeasure?. It is not so difficult, just follow theorganization policies, procedures or controls applicable,and find out what related issues can be measured. Forexample, if one of the policies specifies "An Antivirustool approved by the Security Manager must be installedand running on all computers, and daily actualized", theorganization may decide to measure the following inrelation with this concrete policy:

1. Number of Antivirus tools approved by theSecurity Manager

2. Number of personal computers with antivirusinstalled

3. Number ofservers with antivirus installed4. Number of actualizations distributed in the last

two weeks

In order to use concrete metrics or an indicators aspossible candidates to contribute to the measurementprocess, there are some issues to be considered:

1. It should provide quantifiable and "measurable"results.

2. It has to be based on data available in theorganization.

3. It has to be obtained through processes orprocedures that can be repeated with similar orcomparable results.

4. It should to be possible to keep them independentof the technological platform.

Therefore the information security metrics and indicatorsshould be described, such as to be able to collect, processand present the information. For example the followingitems could be defined for each element:

1 This example will be followed through the rest ofthedocument. I have been asked several limes about "Ok, I canmeasure thingsfrom events, firewa/Is, backups, viruses, etc. butHow do I measure for examp/e management commitment?that's why I did choose this example.

1. Name2. Owner3. Audience4. How the metric or indicator calculates or adds5. The units and manner of representation6. Weight of the concrete metric or indicator within

the measure7. Periodicity8. Thresholds

5.3 Selecting general metrics or indicators

These metrics and indicators should provide general dataor information about the organization, which by itself isnot relevant, but it does if it is combined with otherinformation security metrics.

Some examples of general metrics and indicators might bethe following:

1. Areas or departments2. Employees3. Managers4. Processes5. PCs6. Servers7. Organization budget8. Invoicing9. Customers10. Suppliers

5.4 Selecting management metrics

These metrics should provide data on the organization'sefforts in managing issues such as awareness, legalrequirements, resource allocation, purchasing, etc.

Some examples of management metrics that may be usedto provide data on the previous example of Managementcommitment to infonnation security might be thefollowing:

1. Areas or departments represented in the securitycommittee in the last 6 months

2. Managers attending the security committeemeetings in the last 6 months

3. Managers with a non disclosure agreementsigned

4. Employees with exclusive dedication toinformation security in the last 3 months

5. Information security budget in the current year

5.5 Selecting operation metrics

Operation metrics provide management with data on theperformance of the organization's operations carried outwith personnel intervention.

Some examples of operation metrics that may be used tomeasure the previous example might be the following:

1. Processes affected by the information securitypolicy

2. Incidents reported by managers in the last 3months

3. Incidents assessed in the last 3 months4. Managers with owner responsibilities on

information system assets5. Managers involved in the information security

policy definition, evaluation and review.6. Improvement actions approved in the last 3

months7. Information security related changes approved in

the last 3 months

5.6 Selecting technological metrics

Technological metrics provide management with data onthe performance of the organization's operations carTiedout without personnel intervention.

Some examples of technological metrics that may be usedto measure the previous example might be the following:

1 Manager accounts with passwords solved by abrute force attack in less than 1 hour in themonthly password quality audit

2. Managers monitored in the last 6 months3. Megabytes attached to emails sent from

management accounts m the last month

5.7 Selecting environmental metrics andindicators

Environmental metrics and indicators provide data andinformation about the local, regional, national or globalcondition of the information security.

These category of metrics and indicators provide usefulinformation on relationships between the condition of theenvironment and the an organization's activities.

Some examples of environmental metrics and indicatorsthat may be used to measure the previous example mightbe the following:

1. New legal or regulatory information securityrequirements applicable to the organization'smanagement in the last 12 months

2. Government agencies, academic institutions andnon-governmental organizations reports andstudies related

5.8 Selecting personnel training metrics

Personnel training metrics should provide managementwith data on the personnel qualification.

Some examples of personnel training metrics that may beused to measure the previous example might be thefollowing:

1.2.

Managers with information security certificationManagers which has attended the organizationinformation security training and/or awarenessprogram in the last 12 months

6 REPORTING THE RESULTS

6.1 Collecting metrics

All the selected metrics should be periodically collected toprovide actualized input, but the organization also has tokeep in mind that the metrics collection procedures shouldensure its reliability.

The organization may use its own data or data from othersources. For example, data can be collected from:

1.2.3.4.5.

6.7.8.9.10.11.12.

MonitoringInterviews and observationsRegulatory reportsInventory and production recordsOther management systems running in theorganization (e.g. Quality or Enviromental)Information security products and services logsRisk assessment processInformation security audit or assessment reportsInformation security training recordsReports and studiesCustomers and stakeholdersInformation security associations

6.2 Preparing indicators

The collected metrics should be analyzed and convertedinto information describing the organization's informationsecurity performance.

Metrics assessment may include consideration of the dataquality, validity, adequacy, and completeness necessary toproduce reliable information.

The information describing the organization's informationsecurity performance can be developed using calculations,best estimates, statistical methods and/or graphicaltechniques, or by indexing, aggregating or weighting themetrics registered.

Adding, average, deviation, median, slopes, probabilities,percentages, etc. According to the nature of the metrics orthe indicator being considered the organization shoulddecide which calculation adapts best to what is to beobtained.

Some examples of indicators which combine metrics thatmay be used to measure the previous example might bethe following:

1. II: Relation between the departments representedin the security committee and the totaldepartments

2. I2: Relation between the managers attending thesecurity committee and the total number ofmanagers

3. I3: Relation between the information securitybudget and the total budget

4 I4: Relation between the improvement actionsapproved and the total number of incidentsreported

5. I5: Relation between the managers which hasattended the awareness and training program andthe total number of managers

6. I6: Relation between the managers monitoredand the total number of monitoring processestaken in the last 6 months

Higher level indicators may by defined combining theseindicators, for example the organization may decide thatthe Management commitment to information security isrepresented as:

2x1 + 3xI2 + I3 + I4 + 2xI5 + I6Ia= ---------------------------0o-

10

There is no magical formula to establish the perfectindicator, each organization shall determine whichindicators are useful according to their business and howto get to the results.

6.3 Assessing indicatorsManagement should carefully balance the following twokey issues:

Management should establish the organization criteriawith which the information obtained should be compared.This comparison may indicate progress or deficiencies inthe information security performance, and will help tounderstand why the information security performancehave been met or not.

A key success factor at this point for the organization it toclearly establish the thresholds for each indicator,specifying the concrete actions to take if the indicatorvalue oveaTides the thresholds.

* The stakeholders has information whichdescribes the real situation of the organization'sinformation security, and therefore may help inthe awareness about information security from amore real and objective situation.

a The possible negative impact in the image of theorganization if the results are not as good as itwas expected.

For example, for the indicators II and I2 describedpreviously, the following thresholds could be established:

a Below 50 %:o The meeting takes placeo It will be registered

* Below 33 %:o The meeting does not takes placeo It will be reported as an incident

6.4 Communicating the results

These information has to be reported to management, sothat they have a more objective perspective of howinformation security is going on, and to supportappropriate related actions.

After the results have been analyzed, management shouldensure that appropriate information describing theinformation security performance is communicatedthroughout the organization. This may assist personneland contractors in a better understanding of theirresponsibilities.

Obviously this communication should be a reliablerepresentation of the organization's information securityperformance, but it shall not include information whichmay compromise the organization information security.

It is a common practice for many organizations tocommunicate positive results and tries to hide the negativeones, specially when the organization chooses tocommunicate the results (or an abstract of them) tostakeholders or any other external party.

At this point, the information to be delivered, as well asthe reporting and communicating methods selected,should be carefully assessed by the organization.