© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CATERPILLAR CONFIDENTIAL: GREEN
Automating Malware Beaconing Case Management in a Large Enterprise
Joe Zacharias Manager – Security Architecture
Global Information Services
HP Protect 2014
CATERPILLAR CONFIDENTIAL: GREEN
Agenda
• ArcSight Journey
• Use Case – Malware Beaconing
• Additional Use Cases (Hunting, Investigations, etc.)
• Lessons Learned & Roadmap
• Q&A
CATERPILLAR CONFIDENTIAL: GREEN
Weapon of Mass Construction at the White House !
CATERPILLAR CONFIDENTIAL: GREEN
• Problem Statement • ArcSight Solutions • Source Integration
ArcSight Journey
CATERPILLAR CONFIDENTIAL: GREEN
Problem Statement – SIEM Needs Meet the immediate need for threat management and CSIRT
investigation support • SIEM will provide a solution that enables the CSIRT to detect and respond to
malicious events through real-time monitoring and threat intelligence Provide capability for proactive, behavior-based monitoring with
contextualized alerting • Develop refined use cases, processes, technologies and alerts to detect attacks early
in the attack life cycle; investigating behaviors while reducing false positives Build a framework for a security operations center (SOC)
• Develop people, process and technology around a SIEM that can be scaled and matured for integration into a future SOC
6
Event Legacy Process ArcSight
Event Notification Manual, up to 24 hour delay
Fully automated, near real time
Event Lookup/Research Manual, multi-source correlation
Fully automated, single source
CATERPILLAR CONFIDENTIAL: GREEN
Legacy Malware Process • Batch Load of Firewall Events – Up to 24 hr delay • Manual event lookup to identify malicious host across multiple event
sources by Analyst • ~100 Million Events/Day
ArcSight Integrated Malware Response Use Case
• Real-time FW, Proxy, Antivirus Events • Automated Event Lookup and Correlation • Pivot capability to multiple event sources in single system (ESM) • ~800 Million Events/Day
ArcSight – Where We Were, Where We’re Going
7
CATERPILLAR CONFIDENTIAL: GREEN
Hardware • HP DL580 Gen 7
– 32 Cores, 128 GB RAM
Software • RHEL 6.2 running ESM CORR 6.5 Database
– 90 days online storage
Performance • Currently processing 10,000 -12,000 Events Per Second
ArcSight Solution – ESM Capabilities
CATERPILLAR CONFIDENTIAL: GREEN
Fusion I-O Drives: • Processing 300 million to 800 million events a day • 19,303,919,987 events over past 30 days • 90 days of events stored in ESM • Used only 48% of storage • 6 Fusion I-O Duos RAID configuration • RAID has withstood hard crashes and continued to run ESM while
resyncing, rebuilding spares, or recovering
ArcSight ESM
9
CATERPILLAR CONFIDENTIAL: GREEN
Big Data • 500 Device Sources • ~800 Million Events/Day
– ~12,000 EPS, 36 Million/Hr peak business hrs.
• ~78,000 Rules Firing Each Hour – ~8 Events of Interest Per Hour out of 36M/hr
ArcSight Solution – By the Numbers
10
CATERPILLAR CONFIDENTIAL: GREEN
ArcSight – Hierarchical Architecture
CATERPILLAR CONFIDENTIAL: GREEN
Administration • Connector Health • Logger and ESM System Content Development • Device Source Fidelity and Workflow (DCs, FW,
Proxy, DHCP, etc.)
Management: • MetaNet/EY Advanced Use Case Development
– Custom flex configuration of connectors (CWS, etc.) – Help Desk Use Case Monitoring
• Content Development (OneNote) – Use Case Driven Results of Event Output – Case generation, Alert criteria defined prior to
development
ArcSight – Administration / Management
CATERPILLAR CONFIDENTIAL: GREEN
• Design/Workflow Development • Content Development • Use Case Management
Malware Use Case Development
CATERPILLAR CONFIDENTIAL: GREEN
Malware Beaconing – High Level Overview
Case Management Alerts Dashboards Reports
CATERPILLAR CONFIDENTIAL: GREEN
Malware Use Case – Intelligent Risk-Level Escalation
15
CATERPILLAR CONFIDENTIAL: GREEN
Daily Commodity Malware Response: • Case Management • Integration with Analyst Tools • Intelligent Risk-Level Escalation • Automated Metrics Identification: ArcSight rules identify match to threat intel list and proxy, firewall traffic with instant correlation to DHCP host information Containment: Use integration commands to initiate an automated analysis of machine Recovery: Once the Analyst has contained and recovered the system through normal response process, use case management tools to mark completion, metrics are generated automatically
Malware Use Case – Automation/Integration
16
CATERPILLAR CONFIDENTIAL: GREEN
Daily Commodity Malware Response: • Case Management Metrics • Hot spots
SEP Health Metrics: • Virus Found and Not Cleaned • Top Malware Events in Last 7 Days System Metrics: • Event Throughputs • Hourly / Daily / Monthly Trends • System Health
Dashboards / Metrics
17
CATERPILLAR CONFIDENTIAL: GREEN
• Investigations • VPN Session Tracking • Hunting
New Functionality
CATERPILLAR CONFIDENTIAL: GREEN
Investigations Use Case • Identify Internet Usage by machine with capability to identify average time
spent and/or categorization usage
Help Desk Monitoring Use Case • Monitoring Real-time Windows and LDAP Account Management Events
– Account Disables, Group Management, Lockouts – Trend Analysis – Elevated Privileged Account Creation with Subsequent Deletion
Symantec Antivirus Client Health Use Case • Identify systems with viruses found, but not cleaned
ArcSight – New Functionality
19
CATERPILLAR CONFIDENTIAL: GREEN
Develop Session Use Case Information Identify Session Duration, Geo-Location, and Alerting
VPN Session Tracking
20
CATERPILLAR CONFIDENTIAL: GREEN
VPN Session Tracking – Mapping
CATERPILLAR CONFIDENTIAL: GREEN
Call and email on 1/7/2014 at 10:00 AM involving worm spreading malware on NAS shares: \\arwdfsp01.mw.na.cat.com\shares\Skypee (files owned by internal Caterpillar Employee) Identification: Use ArcSight to identify additional intelligence (host, beacon, spread patterns) Containment: Place device blocking control policy on SEP clients until virus definitions are published. Verify implementation of containment measure with ArcSight Recovery: Use ArcSight to visualize deployment of virus definition and breadth of infected devices across the network
Hunting – Worm Propagation
22
CATERPILLAR CONFIDENTIAL: GREEN
ArcSight 6.0 6.5 Rules Engine – Lightweight Rules Future Roadmap
Lessons Learned
23
CATERPILLAR CONFIDENTIAL: GREEN
Upgrade Issues with 6.5 • Active Lists not updating consistently • Rules not firing when under peak event loads • Console sessions freeze when the Manager service becomes busy Resolutions
• Converted several high firing rules from standard to light • Worked with HP support:
– Tweaked MYSQL settings – Cleaned up internal MySQL issues based on the resvalidate report
• Ordered more disk, RAM, and CPUs for ESM server • Nightly reboot of the ESM server to reset RHEL and ESM
ArcSight 6.0 ArcSight 6.5
24
CATERPILLAR CONFIDENTIAL: GREEN
Issues with Standard Rules • DHCP Sessions - High Insertion Rates
– DHCP active lists would not update consistently
Resolution – Lightweight Rules • Reduced the DHCP events to be only event types that the DHCP rules
were set to fire on • Optimized Infoblox DHCP parser • Converted the DHCP rules for Infoblox and MS DHCP to lightweight
rule set
ArcSight Rules Engine
25
CATERPILLAR CONFIDENTIAL: GREEN
Issues with Hardware/Software • Hourly flushes of RAM to Disk to free up RAM • ESM Crashes
– After several days of continuous operation the ESM server will crash when 99% of the CPU is being used by the Manager and MySQLD for periods of time
Resolutions • What have we done:
– Restarting ESM services daily to rebooting server daily • Actions Planned:
– Add additional CPUs and RAM – Switch ESM from 1Gb network to 10Gb fiber network connection – Upgrade RHEL from 6.2 to 6.4 or latest supported – Upgrade Fusion I-O drivers – Upgrade ESM from 6.5 to 6.5sp1 or latest – Work with HP Professional services to review current Architecture
ArcSight ESM
26
CATERPILLAR CONFIDENTIAL: GREEN
ArcSight Journey – Essential Considerations • Infrastructure: SIEM deployment, integration, and process
development along with professional services assistance
• Staffing: 2 additional staff (ArcSight Engineers): – Rules/Content Developer: Create and maintain the SIEM content ranging from full spectrum
content development and ongoing maintenance – Device Source Analyst: Maintain all existing and new integration sources, architecture updates,
and connector troubleshooting
• Training: Train first and second level responders on ArcSight consoles then train engineers for connector, database, and use case support
• Maintenance: Maintain ongoing engagement with professional services for annual health checks and use case development
27 Prepared by Joe Zacharias
CATERPILLAR CONFIDENTIAL: GREEN
ArcSight Journey – Roadmap
3 Phases to SIEM Integration 1. SIEM Implementation (Year 1)
– Deployment, integration, development – Professional services – Staffing/Training
2. SIEM Maturation (Year 2-3) – Advanced use case integration – Streamline people & processes – Plan to expand architecture
3. SOC Integration (Year 4-6) – 24x7 SOC operations
• Virtual, Follow the Sun, or Co-located – Deployment, staffing, integration
28 Prepared by Joe Zacharias
CATERPILLAR CONFIDENTIAL: GREEN
Q & A
Questions?
29
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 30
Use the mobile app 1. Click on Sessions 2. Click on this session 3. Click on Rate Session
Or use the hard copy surveys
Thank you for providing your feedback, which helps us enhance content for future events.
Session TB3088 Speaker Joe Zacharias
Please give me your feedback
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Top Related