© Copyright 2014 Hewlett -Packard Development Company, …Automating Malware Beaconing Case...

CATERPILLAR CONFIDENTIAL: GREEN

Automating Malware Beaconing Case Management in a Large Enterprise

Joe Zacharias Manager – Security Architecture

Global Information Services

HP Protect 2014


Agenda

• ArcSight Journey

• Use Case – Malware Beaconing

• Additional Use Cases (Hunting, Investigations, etc.)

• Lessons Learned & Roadmap

• Q&A


Weapon of Mass Construction at the White House !


• Problem Statement • ArcSight Solutions • Source Integration

ArcSight Journey


Problem Statement – SIEM Needs Meet the immediate need for threat management and CSIRT

investigation support • SIEM will provide a solution that enables the CSIRT to detect and respond to

malicious events through real-time monitoring and threat intelligence Provide capability for proactive, behavior-based monitoring with

contextualized alerting • Develop refined use cases, processes, technologies and alerts to detect attacks early

in the attack life cycle; investigating behaviors while reducing false positives Build a framework for a security operations center (SOC)

• Develop people, process and technology around a SIEM that can be scaled and matured for integration into a future SOC

6

Event Legacy Process ArcSight

Event Notification Manual, up to 24 hour delay

Fully automated, near real time

Event Lookup/Research Manual, multi-source correlation

Fully automated, single source


Legacy Malware Process • Batch Load of Firewall Events – Up to 24 hr delay • Manual event lookup to identify malicious host across multiple event

sources by Analyst • ~100 Million Events/Day

ArcSight Integrated Malware Response Use Case

• Real-time FW, Proxy, Antivirus Events • Automated Event Lookup and Correlation • Pivot capability to multiple event sources in single system (ESM) • ~800 Million Events/Day

ArcSight – Where We Were, Where We’re Going

7


Hardware • HP DL580 Gen 7

– 32 Cores, 128 GB RAM

Software • RHEL 6.2 running ESM CORR 6.5 Database

– 90 days online storage

Performance • Currently processing 10,000 -12,000 Events Per Second

ArcSight Solution – ESM Capabilities


Fusion I-O Drives: • Processing 300 million to 800 million events a day • 19,303,919,987 events over past 30 days • 90 days of events stored in ESM • Used only 48% of storage • 6 Fusion I-O Duos RAID configuration • RAID has withstood hard crashes and continued to run ESM while

resyncing, rebuilding spares, or recovering

ArcSight ESM

9


Big Data • 500 Device Sources • ~800 Million Events/Day

– ~12,000 EPS, 36 Million/Hr peak business hrs.

• ~78,000 Rules Firing Each Hour – ~8 Events of Interest Per Hour out of 36M/hr

ArcSight Solution – By the Numbers

10


ArcSight – Hierarchical Architecture


Administration • Connector Health • Logger and ESM System Content Development • Device Source Fidelity and Workflow (DCs, FW,

Proxy, DHCP, etc.)

Management: • MetaNet/EY Advanced Use Case Development

– Custom flex configuration of connectors (CWS, etc.) – Help Desk Use Case Monitoring

• Content Development (OneNote) – Use Case Driven Results of Event Output – Case generation, Alert criteria defined prior to

development

ArcSight – Administration / Management


• Design/Workflow Development • Content Development • Use Case Management

Malware Use Case Development


Malware Beaconing – High Level Overview

Case Management Alerts Dashboards Reports


Malware Use Case – Intelligent Risk-Level Escalation

15


Daily Commodity Malware Response: • Case Management • Integration with Analyst Tools • Intelligent Risk-Level Escalation • Automated Metrics Identification: ArcSight rules identify match to threat intel list and proxy, firewall traffic with instant correlation to DHCP host information Containment: Use integration commands to initiate an automated analysis of machine Recovery: Once the Analyst has contained and recovered the system through normal response process, use case management tools to mark completion, metrics are generated automatically

Malware Use Case – Automation/Integration

16


Daily Commodity Malware Response: • Case Management Metrics • Hot spots

SEP Health Metrics: • Virus Found and Not Cleaned • Top Malware Events in Last 7 Days System Metrics: • Event Throughputs • Hourly / Daily / Monthly Trends • System Health

Dashboards / Metrics

17


• Investigations • VPN Session Tracking • Hunting

New Functionality


Investigations Use Case • Identify Internet Usage by machine with capability to identify average time

spent and/or categorization usage

Help Desk Monitoring Use Case • Monitoring Real-time Windows and LDAP Account Management Events

– Account Disables, Group Management, Lockouts – Trend Analysis – Elevated Privileged Account Creation with Subsequent Deletion

Symantec Antivirus Client Health Use Case • Identify systems with viruses found, but not cleaned

ArcSight – New Functionality

19


Develop Session Use Case Information Identify Session Duration, Geo-Location, and Alerting

VPN Session Tracking

20


VPN Session Tracking – Mapping


Call and email on 1/7/2014 at 10:00 AM involving worm spreading malware on NAS shares: \\arwdfsp01.mw.na.cat.com\shares\Skypee (files owned by internal Caterpillar Employee) Identification: Use ArcSight to identify additional intelligence (host, beacon, spread patterns) Containment: Place device blocking control policy on SEP clients until virus definitions are published. Verify implementation of containment measure with ArcSight Recovery: Use ArcSight to visualize deployment of virus definition and breadth of infected devices across the network

Hunting – Worm Propagation

22


ArcSight 6.0 6.5 Rules Engine – Lightweight Rules Future Roadmap

Lessons Learned

23


Upgrade Issues with 6.5 • Active Lists not updating consistently • Rules not firing when under peak event loads • Console sessions freeze when the Manager service becomes busy Resolutions

• Converted several high firing rules from standard to light • Worked with HP support:

– Tweaked MYSQL settings – Cleaned up internal MySQL issues based on the resvalidate report

• Ordered more disk, RAM, and CPUs for ESM server • Nightly reboot of the ESM server to reset RHEL and ESM

ArcSight 6.0 ArcSight 6.5

24


Issues with Standard Rules • DHCP Sessions - High Insertion Rates

– DHCP active lists would not update consistently

Resolution – Lightweight Rules • Reduced the DHCP events to be only event types that the DHCP rules

were set to fire on • Optimized Infoblox DHCP parser • Converted the DHCP rules for Infoblox and MS DHCP to lightweight

rule set

ArcSight Rules Engine

25


Issues with Hardware/Software • Hourly flushes of RAM to Disk to free up RAM • ESM Crashes

– After several days of continuous operation the ESM server will crash when 99% of the CPU is being used by the Manager and MySQLD for periods of time

Resolutions • What have we done:

– Restarting ESM services daily to rebooting server daily • Actions Planned:

– Add additional CPUs and RAM – Switch ESM from 1Gb network to 10Gb fiber network connection – Upgrade RHEL from 6.2 to 6.4 or latest supported – Upgrade Fusion I-O drivers – Upgrade ESM from 6.5 to 6.5sp1 or latest – Work with HP Professional services to review current Architecture

ArcSight ESM

26


ArcSight Journey – Essential Considerations • Infrastructure: SIEM deployment, integration, and process

development along with professional services assistance

• Staffing: 2 additional staff (ArcSight Engineers): – Rules/Content Developer: Create and maintain the SIEM content ranging from full spectrum

content development and ongoing maintenance – Device Source Analyst: Maintain all existing and new integration sources, architecture updates,

and connector troubleshooting

• Training: Train first and second level responders on ArcSight consoles then train engineers for connector, database, and use case support

• Maintenance: Maintain ongoing engagement with professional services for annual health checks and use case development

27 Prepared by Joe Zacharias


ArcSight Journey – Roadmap

3 Phases to SIEM Integration 1. SIEM Implementation (Year 1)

– Deployment, integration, development – Professional services – Staffing/Training

2. SIEM Maturation (Year 2-3) – Advanced use case integration – Streamline people & processes – Plan to expand architecture

3. SOC Integration (Year 4-6) – 24x7 SOC operations

• Virtual, Follow the Sun, or Co-located – Deployment, staffing, integration

28 Prepared by Joe Zacharias


Q & A

Questions?

29

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 30

Use the mobile app 1. Click on Sessions 2. Click on this session 3. Click on Rate Session

Or use the hard copy surveys

Thank you for providing your feedback, which helps us enhance content for future events.

Session TB3088 Speaker Joe Zacharias

Please give me your feedback


Thank you

© Copyright 2014 Hewlett -Packard Development Company, …Automating Malware Beaconing Case...

Documents

Transcript of © Copyright 2014 Hewlett -Packard Development Company, …Automating Malware Beaconing Case...