Towards Total Security Quality Management (TSQM): Enterprise Perception Measurement and the “House...

Towards Total Security Quality Management (TSQM):

Enterprise Perception Measurement and the “House of Security”

February 16, 2006 Professor Stuart Madnick, Dr. Michael Siegel,

Wee Horng Ang ({smadnick, msiegel, weeang}@mit.edu)

Copyright © 2006, MIT 2

FACULTY

• Yang Lee

• Stuart Madnick

• Michael Siegel

• Diane Strong

• Richard Wang

• Chrisy Yao

STUDENTS

• Wee Horng Ang

• Dinsha Mistree

• Venkataramana Thummisi

MIT TEAM


Academic Literature

Overview of Project ComprehensiveList of Aspects

of Security

Industry Literature

Key DimensionsAnd Aspects

Stakeholders And Roles

Survey1 and 2

Survey 3

Gap AnalysisInstrument

Key GapFindings

Gap Analysis


Validation and Refinement


Brief Description of SurveysSurvey 1

Open-ended: What does holistic Security mean to you?

Survey 2Semi-structured: What does holistic Security mean to you? Similar to Survey 1, but starts with 20 security aspects.

Survey 3 13 semi-structured questions regarding Extended Enterprise security covering issues such as Security Return on Investment, Benefits of Security, and Extended Enterprise Security.


Comprehensive List of Aspects of Security

Ability to effectively use data

acceptance inspection

Access

access control mechanism

access level

access list

Access modes

access period

access port

access type

Accountability

accreditation

accreditation authority

add-on security

administrative security

Alert handling

Antivirus

Asset classification & control

assurance

attack

audit trail

authenticate

Authentication

authenticator

authorization

automated information system (AIS)

automated information system security

automated security monitoring

availability of data

Availability of service

back door

backup plan

Bell-La Padula model

benign environment

between-the-lines entry

Brand equity “is tied to customer’s perception about security

Breach of confidentiality Breach of Security (BOS)

Breach of integrity (BOI)

browsing

Buffer overflow

Business loss

Cache overflow

call back

capability

category

certification

closed security environment

communications security (COMSEC)

Company preparedness

compartment

compartmented security mode

Competitive edge

Compliance

compromise

compromising emanations

computer abuse

computer cryptography

computer fraud

computer security subsystem

concealment system

confidentiality

configuration control

configuration management

confinement

confinement channel

confinement property

Connection

contamination

contingency plan

control zone

controlled access

controlled sharing

Controls

Cookies

cost-risk analysis

countermeasure

covert channel

covert storage channel

covert timing channel

Credibility

Criteria

crypto-algorithm

Cryptosecurity

Customer confidence

Customer loss

Customers system

Customized access

Data control

Data encryption

Data Encryption Standard (DES)

Data reliability

dedicated security mode

default classification

Degausser Products List

Denial of Service

… AND MANY MORE …


Academic Literature

Overview of Project – Key Dimensions

ComprehensiveList of Aspects

of Security

Industry Literature



Survey1 and 2

Survey 3


Key GapFindings

Gap Analysis




Dimensions of Security

TechnologyResources

For Security

FinancialResources

ForSecurity

Business Strategy

ForSecurity

SecurityPolicy &

Procedures

SecurityCulture

Accessibility Confidentiality

Vulnerability

“House of Security”


Good SecurityGood Security provides Accessibility to data

and networks to appropriate users while simultaneously protecting Confidentiality of data and minimizing Vulnerabilities to attacks and threats.

Good Security Practice goes beyond technical IT solutions. It is driven by a Business Strategy with associated Security Policies and Procedures implemented in a Culture of Security. These are practices are supported by IT Resources and Financial

Resources dedicated to Security.


Academic Literature

Overview of Project – Stakeholders and Roles


of Security

Industry Literature



Survey1 and 2

Survey 3


Key GapFindings

Gap Analysis




Stakeholders

Enterprise

General Public

Extended Enterprise

Ring 1: EnterpriseRing 2: Extended EnterpriseRing 3: General Public


Stakeholders & RolesDomain/Role

Level/Rank General business

IT Organization General security/ physical security

Partners (Extended Enterprise)

Top exec CEO, CFO, … Top IT Mgt/CIO Top Security Mgt / CSO

Line/middle manager

Business unit manager

IT non-security managers -------------------------IT security manager

Security managers

Workers Business personnel

IT non-security personnel -------------------------IT security personnel

Security personnel (e.g., guard)


Academic Literature

Overview of Project – Gap Analysis


of Security

Industry Literature



Survey1 and 2

Survey 3


Key GapFindings

Gap Analysis




Differing Perceptions

Picture of old lady or young lady ?

Perceptions are as important as “reality”


Data Source(How do you cite in a Journal article?)


Purpose of Gap Analysis

Purpose of Gap Analysis is to understand Differences in Perceptions between factors such:

(A) Security Status Assessment and Security Importance

(B) views of diverse Security Stakeholders

…within the Enterprise and across the Extended Enterprise


Purpose of Gap Analysis (cont.)

Gaps represent Opportunities for Improvement within the Enterprise and across the Extended Enterprise

(A) When Status is below the Needs, these represent Areas for Improvement

(B) When Status among Stakeholders show differences, these represent areas for Investigating sources of the differences• Gaps may represent misunderstandings• Gaps may represent differences in local

knowledge and needs


Many Types of Gaps1. Performance Gaps: Current Status v.

Importance2. Role Gaps: e.g., Business Managers v. IT

staff3. Inter-Enterprise Gaps: Internal Line

Manager v. Supplier

Initially, our focus is on Performance Gaps, much more data needed for analyzing Role and Enterprise Gaps

Issue: Gathering of enough data from same organization and partner data


Gap Analysis Questionnaire

1. Questionnaire respondents are comprised of the diverse roles (IT, IT security, Users, Business managers, Executives, etc.) within the enterprise and across (suppliers, customers, collaborators, etc.) the extended enterprise.

2. Each respondent reports his/her view of

actual assessment and importance of each

aspect for both his/her organization and a

partner organization.


Gap Analysis Questionnaire (cont.)

3. Questions on the questionnaire cover the 8 constructs of security:• Accessibility• Vulnerability• Confidentiality• Financial resources for security• Technology resources for security• Business strategy for security• Security policy and procedures• Security culture

4. To ensure construct validity, (approx) 5 questions are included for each construct.


Extended Enterprise Security SurveyForm # 01-23

Towards Total Security Quality Management (TSQM) MIT’s Extended Enterprise Security Survey

Introduction The following survey is part of a research project at MIT to develop a holistic framework to study enterprise

security within and between organizations. Your responses to the following survey will provide us valuable insight about extended enterprise security. The extended enterprise includes an organization and its suppliers, customers, partners, and competitors. Extended enterprise security is concerned with security both within and between these organizations.

The survey should take you about 20 minutes to fill out.

Note about confidentiality: Your responses to questionnaire items will not be revealed to your organization or to any other organization. Only aggregate results will be used in our analyses. If you would like to receive a copy of our research results, please provide your email address at the bottom of the survey.

General Instructions1. What does it mean by “assessment” and “importance”?The survey asks you to give your impression of the “assessment” and “importance” of various security issues. “Assessment,” means your view of how well your organization is doing on these issues. “Importance” means your view of how important this issue is to you.

2. There is no right or wrong answer to any question. We are asking for your view. You may not know exact details about your company’s security. We are not asking for these details, but asking for

your views. Please give your best estimate.

3. What is “Partner Organization”?The survey also asks you to give your impressions of “assessment” and “importance” for ONE partner

organization. This partner organization should be one of your suppliers, if feasible. Alternatively, please select a customer or a collaborator organization.

4. There is no right or wrong answer about a partner’s security. We are asking your views of the partner organization’s security, you do not need to know exact details. Please

give your best estimate. If you have no knowledge at all of an aspect of your partner security, you may leave that question blank.

Thank you, MIT TSQM team


Your Organization & PartnerExtended Enterprise Security SurveySection 1: Your OrganizationYour Organization/Company Organization Name__________________________________________________________Industry____________________________________________________________________Approximate total number of employees in your entire organization: ________________Your Job Title and Work Role ___________________________________________________________________________________________________________________________Department/Division/Group___________________________________________________In my organization, I am a: _____(1) Executive (CEO,CFO, VP etc.) _____(2) Functional or Line Manager_____(3) Professional (Consultant, Engineer, In-house Expert, etc.) _____(4) Other Organizational Member In my organization, I work in the area of:_____(1) Business Security Policy and Management_____(2) IT Security _____(2) IT but not in Security, _____(3) General/Physical Security, _____(4) Not in Security or in IT.Section 2: Your Partner OrganizationPick one partner organization for answering these questions. The survey administrator may give you additional instructions about picking a partner

origination. All answers about your partner organization should be about ONE specific organization.Your Partner Organization/Company Partner Organization’s Name (optional)__________________________________________Partner’s Industry_____________________________________________________________Approximate total number of employees in your partner organization: ________________Your Partner Organization is your organization’s: _____(1) Supplier ____(2) Customer ____(3) Collaborator ____(4) CompetitorMajor Group/Division/Department you usually work with:_______________________________________________________________________


Security Questions (40)


Survey Data Gathering• Developed web-based survey• Developed secure (https) web-based

survey instrument• Collected data

– Considerable “partner” company data, but need more

– Both “miscellaneous” and several company-wide

• Valuable for intra-company stakeholder gap analyses

• Preliminary analysis of increased pilot data– Some sample analysis follows …


Lots of Survey Data Gathered


Academic Literature

Overview of Project – Key Findings


of Security

Industry Literature



Survey1 and 2

Survey 3


Key GapFindings

Gap Analysis




Gap Analysis Preliminary Findings

Mostly Performance GapsSome Role and Inter-Enterprise Gaps

Explore: at item level (yet not construct level)- Data recently received- Only very limited analysis so far

- All Findings that follow are preliminary


Some speculation:Sample Security Culture

• Assessment vs Importance? AssessmentImportance About same (10%)

?? ?? ??

• Assessment: Your Organization vs PartnerYour Org Partner About Same

(10%)

?? ?? ??

• Assessment: Different Roles/FunctionsIT Security IT, not Security Gen’l Mgt

?? ?? ?? ?? ?? ?? (lowest?)

Question 39: People are aware of good security practices.


Evaluating Statistical Significance

Significant at 99.99% level

28

Significant at 99% level

11


0


1

Less than 90% 0

Total 40

Gap significance notation:*** Significant at the 99.99% level; ** Significant at the 99% level; * Significant at the 95% level; ~ Significant at the 90% level.

MA vs MI Gaps:


Question 18: People in the organization carefully follow good security practices.

Question 26: People in the organization can be trusted not to tamper with data and networks.

Question 39: In the organization, people are aware of good security practices.

Qn 18 Gap = 1.24 ***

Qn 26 Gap = 1.01 ***

Qn 39 Gap = 1.28 ***

Gap Analysis FindingsSecurity Culture

4 5 6 7

MA

MI

MA

MI

MA

MI

Qn 39Qn 39 GapQn 26Qn 26 GapQn 18Qn 18 Gap


Gap Analysis Findings – Different Organizations

Gap between Assessment and Importance – for your company

Overall = 1.28 (5.04 vs. 6.32)

Miscellaneous 1 = 2.40 (4.20 vs. 6.60)

Company X 2 = 1.83 (5.00 vs. 6.83)

Company W 2 = 1.89 (4.61 vs. 6.50)

Company I 3 = 0.44 (5.33 vs. 5.78)

2 High-tech organizations3 Non-USA company


1 Original pilot sample: diverse array of companies many middle-managers

MA Gap

MI


Gap Analysis Findings – Compared with Partner

Organization

Gap between Assessment and Importance – for your company

Overall = 1.28 (5.04 vs. 6.32)

Gap between Assessment and Importance – for partner company

Overall = 0.70 (5.25 vs. 5.95)


General conclusion: - View partner as “better” in assessment - But it is also “less important” -> So Gap is much less

But not exactly true for all organizations …


Gap Analysis Findings – Compared with Partner

Organization

Some observations: * Gaps all smaller, but * Assessment +/-

* Importance +/-


Your Organization Partner Organization

4 5 6 7

Comp I

Comp W

Overall

MA Gap

MI

4 5 6 7

Comp I

Comp W

Overall

PA Gap

PI


Gap Analysis Findings – Different Roles/Areas

Your Organization Partner OrganizationQuestion 39: People are aware of good security practices.

4 5 6 7

NonIT/Security

IT, Non-Security

IT Security

Overall

4 5 6 7

NonIT/Security

IT, Non-Security

IT Security

Overall

Some observations:• Not huge difference in gaps for “your organization” - More significant gaps in views of partner organization• IT Security people perceive much less “gap” in partner - And much lower “importance” for partner


Academic Literature

Overview of Project – Instrument Validation


of Security

Industry Literature



Survey1 and 2

Survey 3


Key GapFindings

Gap Analysis




Phase 2: Underway (mostly completed)

• Collect more data – especially for intra-company (partner) stakeholder analysis

• Complete analysis of pilot data• Complete construct analysis• Refine stakeholder and dimensions• Refine questionnaire items • Revise gap analysis instrument


Instrument Analysis for Construct Reliability and Validity

1. Reliability – means produces consistent results- The multiple questions (components) for each

construct produce strongly correlated responses- Determined by computing Cronbach Alphas

2. Validity – means components are more closely correlated with the others of that construct than they are with components of another construct

- Convergent Validity – form a single construct• Evaluated using Average Variance Extracted (AVE)

- Discriminant Validity – not part of another construct

• Evaluated by requirement that squared multiple correlation between two constructs less than AVE of each construct


Analysis of Construct Reliability and Validity

Cronbach's Alpha MA MI Qn Removed Qn AddedAccessibility 0.695488 0.659241 22Vulnerability 0.592442 0.610124 15Confidentiality 0.660975 0.70541 3, 31 22FinancialResources 0.740053 0.748803 2ITResources 0.640141 0.701367 5BusinessStrategy 0.807072 0.711837 19SecurityPolicy 0.745929 0.671639 25 19SecurityCulture 0.682589 0.706474 9

Construct ValidityAccessibility Vulnerability Confidentiality FinancialResourcesITResources BusinessStrategy SecurityPolicy SecurityCulture

Accessibility 0.64772479Vulnerability 0.47097 0.5384088Confidentiality 0.52719 0.45185 0.616511103FinancialResources 0.35937 0.50057 0.49552 0.76924164ITResources 0.47631 0.62317 0.57952 0.69041 0.62101291BusinessStrategy 0.18955 0.57171 0.42095 0.66325 0.53096 0.863862672SecurityPolicy 0.25447 0.58709 0.53149 0.58769 0.54148 0.72785 0.77396066SecurityCulture 0.24363 0.46202 0.34463 0.58476 0.48422 0.66837 0.5285 0.583653002

In the Construct Validity table, diagonals >0.50 indicates good convergent validity, and having the valuesof the non-diagonal rows and columns of each construct lower than the diagonals indicates good discriminant validity.


Revised InstrumentQn No.

Original Questions

7 The organization checks the identity of users before allowing access to data and networks. ok14 The organization’s data and networks are only available to approved users. ok22 The organization has adequate policies about user identifications, passwords, and access privileges. Move to factor 336 The organization provides access to data and networks to legitimate users. ok40 The organization’s data and networks are usually available when needed. ok

1 The organization’s data and networks are rarely tampered with by unauthorized access. ok8 The organization has adequate safe guards against internal and external threats to its data and networks. ok

15 The organization’s network is rarely unavailable due to attacks (for example, denial of service, hacker break-ins, viruses and worms).

delete

23 The organization improves its security by learning from previous attacks on its data and networks. ok37 The organization has a rapid response team ready for action when attacks occur. ok

3 Customers trust the organization not to disclose data about them. delete16 The organization has adequate policies for when and how data can be shared. ok24 The organization protects privacy of personal data (for example, customer data, data about employees). ok31 This organization has someone who manages the use, storage, and sharing of confidential data. delete38 The organization provides good protection of confidential corporate data. ok22 The organization has adequate policies about user identifications, passwords, and access privileges. From factor 1

2 In the organization, security is adequately funded. delete12 In the organization, security funds are appropriately distributed based on needs. ok20 Security is a funding priority in the organization. ok28 The organization has enough security personnel to cover its security needs. ok34 The organization makes good use of available funds for security. ok

5 Business managers in the organization are involved with IT security policies. delete6 The organization has enough IT security specialists to cover its security needs. ok

13 In the organization, the IT group takes security seriously. ok17 The organization has adequate technology for supporting security ok21 The organization uses its IT security resources effectively to improve security. ok

4 The organization’s security strategy sets direction for its security practices. ok19 The organization has a well-defined and communicated security strategy. Move to factor 727 Security is a business agenda item for top executives in the organization. ok33 In the organization, business managers help set the security strategy. ok35 The organization’s security strategy is well publicized in the organization. ok

10 The organization has policies for regularly-scheduled security audits. ok25 The organization has adequate procedures for ensuring the physical security of buildings and equipment. delete29 The organization has a well-defined policies and procedures for data and network security.. ok30 The organization has procedures for detecting and punishing security violations. ok19 The organization has a well-defined and communicated security strategy. From factor 6

9 The burden of security policies on people in the organization is minimal. delete11 People in the organization are knowledgeable about IT security tools and practices. ok18 People in the organization carefully follow good security practices. ok26 People in the organization can be trusted not to tamper with data and networks. ok32 People in the organization can be trusted to engage in ethical practices with data and networks. ok39 In the organization, people are aware of good security practices. ok

Fac

tor

6:

Bu

sin

ess

Str

ateg

y

Fac

tor

5: IT

R

eso

urc

es

Fac

tor

8:

Sec

uri

ty

Cu

ltu

re

Fac

tor

7:

Sec

uri

ty

Po

licy

Fac

tor

2:

Vu

lner

abili

ty

Fac

tor

1:

Acc

essi

bili

ty

Fac

tor

4:

Fin

anci

al

Res

ou

rces

Fac

tor

3:

Co

nfi

den

tial

ity


Average Construct Values

4.5

5

5.5

6

6.5

7

Accessibility

Vulnerability

Confidentiality

Financial Resources

IT Resources

Business Strategy

Security Policy

Security Culture

PI

PA

MI

MA


Constructs – Average Values & Standard Deviations

Standard Deviation Across Constructs

3

4

5

6

7

8

MA 6.20515 5.662 5.94321 5.30159 5.80196 5.19679 5.38291 5.09089

MI 6.61953 6.32517 6.53759 6.12497 6.36107 5.98732 6.20057 6.23717

PA 6.07735 5.40918 5.76296 5.12257 5.55998 5.01523 5.22793 5.06521

PI 6.41275 6.04624 6.28249 5.67872 5.98215 5.61832 5.6816 5.77504

Accessibility Vulnerability ConfidentialityFinancial

ResourcesIT Resources

Business Strategy

Security Policy Security Culture


Average Construct Variances

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2Accessibility

Vulnerability

Confidentiality

Financial Resources

IT Resources

Business Strategy

Security Policy

Security Culture

Construct Variation MI-MA

Construct Variation PA-MA

Construct Variation PI-PA

Construct Variation PI-MI

Zero Axis


Absolute Construct Variances

0

0.2

0.4

0.6

0.8

1

1.2

Accessibility

Vulnerability

Confidentiality

Financial Resources

IT Resources

Business Strategy

Security Policy

Security Culture

Construct Variation MI-MA

Construct Variation |PA-MA|

Construct Variation |PI-MI|

Construct Variation PI-PA


Some Preliminary Insights 1 Highest assessments in accessibility indicates that businesses are still

primarily concerned with information access and use. Low assessment in security culture, further confirms that security management have yet to mature to the same level of security awareness and depth.

2 Low Gaps in overall Accessibility levels states that accessibility is very well-established, perhaps to the point of saturation.

3 High standard deviations in Security Policy indicates there is a disparity between the various companies/ industries.

4 The large MI-MA gap, and PI -PA gap in security culture, shows companies are beginning to understand the need to achieve further improvement, highlighting an important area of potential growth.

6 Partners assessment lower than self assessment indicates the aura of "invincibility" is present,that companies believe they are safer than their partners. [Of course, everyone is someone elses partner.]

7 Partners importance of security lower than self security reiterates the point that they believe their own companies rate these qualities more importantly on their agenda than would their partners.


Next steps: Phase 3• Large-scale Gap Analysis Study

– IBM– Nortel – RSA Security Conference

• (mailing post-conference)

– 25-50 responses from 3 or more members of eBusiness Center (e.g., BT, UPS)

– Cisco

Two rounds: 500 responses5000 responses

• Extensive Gap Analysis Results


Next steps: Phase 4 (longer-term)

• Longer-term: Pursue other related security measurement activities:– Other Survey Instruments– Case Studies– Best Practices– Benchmarking– Security Methodology


What is “Good Security?”

It can be a matter of opinion (perception)


Thank you

Stuart MadnickT 617-253-6671 E [email protected]://web.mit.edu/smadnick/www/Projects/I-SEE%20CeB.pdf


Extra Slides


Gap Analysis FindingsAccessibility

Question 40: The organization’s data and networks are usually available when needed

Gap = 0.40 **

6.72 (My Importance)

vs.

6.32 (My Assessment)

Note: ** indicates significant at the 99% level5 6 7

MA

MI

AccessibilityMy Org Gap


Gap Analysis FindingsVulnerability

Question 1: The organization’s data and networks are rarely tampered with by unauthorized access.

Gap = 1.22 ***


vs.


Note: *** indicates significant at the 99.99% level

5 6 7

MA

MI

VulnerabilityMy Org Gap


Gap Analysis FindingsConfidentiality

Question 38: The organization provides good protection of confidential corporate data.

Gap = 0.58 ***


vs.


5 6 7

MA

MI

ConfidentialityMy Org Gap


Gap Analysis FindingsFinancial Resource for

SecurityQuestion 2: In the organization, security is

adequately funded.

Gap = 0.78 ***


vs.


5 6 7

MA

MI FinancialResourcesMy Org Gap


Gap Analysis Findings IT Resource for Security

Question 5: Business managers are involved with IT security policies.

Question 17: The organization has adequate technology for supporting security.

Qn 5 Gap = 1.08 ***

5.96 (My Importance) vs.


Qn 17 Gap = 0.51 **

6.37 (My Importance) vs.


4 5 6 7

MA

MI

MA

MI

Qn 17Qn 17 GapQn 5Qn 5 Gap


Question 4: The organization’s security strategy sets directions for its security practices.

Question 19: The organization has a well-defined and communicated security strategy.

Qn 4 Gap = 0.64 *** 6.33 (My Importance) vs.


Qn 19 Gap = 1.07 *** 6.14 (My Importance)

vs. 5.07 (My Assessment)

Gap Analysis FindingsBusiness Strategy for

Security

4 5 6 7

MA

MI

MA

MI



Question 25: The organization has adequate procedures for ensuring the physical security of buildings and equipment.

Question 30: The organization has procedures for detecting and punishing security violations.

Qn 25 Gap = 1.07 *** 6.42 (My Importance) vs.


Qn 30 Gap = 0.94 *** 6.25 (My Importance)

vs. 5.31 (My Assessment)

Gap Analysis FindingsPolicy and Procedures for

Security

4 5 6 7

MA

MI

MA

MI


Towards Total Security Quality Management (TSQM): Enterprise Perception Measurement and the “House...

Documents

Transcript of Towards Total Security Quality Management (TSQM): Enterprise Perception Measurement and the “House...