Network Monitoring & Troubleshooting plus Log

AnalysisFaculty:

Scott Greeneof

Evidence Solutions, Inc.

Scott@EvidenceSolutions.comwww.EvidenceSolutions.com

►10 signs that you aren't cut out for IT 1: You lack patience

10 Signs you aren’t cut out for IT

U of Nebraska Incident

►An Undergrad suspected in Univ. of Nebraska breach where more than 650K personal records were compromised in attack.

►The intrusion was into a university database containing personal information on more than 650,000 students, parents and employees.

U of Nebraska Incident

►The intrusion, which was described by university officials as a "skilled attack," exposed the Social Security Numbers (SSNs), names, addresses, course grades financial aid and other information on students who attended the university since 1985.

U of Nebraska Incident

►The breach also exposed personal data and financial information for parents of students who applied for financial aid at UNL, according to the university.

►A staff member in UNL's Computing Services Network discovered the breach in the Nebraska Student Information System (NeSIS) on May 23.

U of Nebraska Incident

►An Undergrad suspected in Univ. of Nebraska breach where more than 650K personal records were compromised in attack.

►The intrusion was into a university database containing personal information on more than 650,000 students, parents and employees.

U of Nebraska Incident

►The system manages student admissions, campus housing and course registration.

►It was built over a three-year period at a cost of $29.9 million, has been operational for the past two years and is based on Oracle's PeopleSoft Enterprise Campus Solution platform.

U of Nebraska Incident

►An FAQ on the incident posted by the university makes it clear that personal data in the breached server was not encrypted. "However, we are confident that the type of attack we experienced would have bypassed any encryption that was in place," the university said.

U of Nebraska Incident

►The vulnerability that enabled the intrusion has since been closed and the university is currently working with a third-party firm to review and address remaining vulnerabilities.

20 Critical Security Controls

►1) Inventory of Authorized & Unauthorized Devices

►2) Inventory of Authorized & Unauthorized Software

►3) Secure Configurations for Hardware & Software on Laptops, Workstations, & Servers

20 Critical Security Controls

►4) Continuous Vulnerability Assessment & Remediation

►5) Malware Defenses

20 Critical Security Controls

►6) Application Software Security Code Reviews Proper Logging Abnormal operation reporting

20 Critical Security Controls

►7) Wireless Device Control►8) Data Recovery Capability►9) Security Skills Assessment and

Appropriate Training to Fill Gaps►10) Secure Configurations for Network

Devices such as Firewalls, Routers, and Switches

20 Critical Security Controls

►11) Limitation and Control of Network Ports, Protocols, and Services Including custom applications Development departments need to

communicate with network departments

20 Critical Security Controls

►12) Controlled Use of Administrative Privileges

►13) Boundary Defense See 10 Include penetration testing Include review of firewall rules Remote Users Mobile Devices

20 Critical Security Controls

►14) Maintenance, Monitoring, & Analysis of Security Audit Logs

►15) Controlled Access Based on the Need to Know

20 Critical Security Controls

►16) Account Monitoring and Control What do users have rights to

►Why? What do processes have rights to

►Why?

20 Critical Security Controls

►17) Data Loss Prevention►18) Incident Response Capability

Who responds Test those responses Who gets notified

►Hr►Legal

20 Critical Security Controls

►19) Secure Network Engineering aka “Develop a Secure Infrastructure”

►20) Penetration testing

#14

►Validate audit log settings for each hardware device and the software installed on it, ensuring that logs include: Date Timestamp source addresses destination addresses Any other useful information

#14

►Normalize Logs Syslog Common Event Expression initiative Use normalization tools to convert logs

#14

►Reports Security personnel and/or system

administrators should run weekly reports that identify anomalies in logs.

They should then actively review the anomalies, documenting their findings.►A log for the log events

#14

►Time Synch Use at least two synchronized time

sources All servers and network equipment should

be in synch.►Test►Validate their synchness

Federal Security Standards

►NIST Special Publication (SP) 800-37 Categorize

►The information system and the information processed, stored, and transmitted by that system based on an impact analysis

Federal Security Standards

►NIST Special Publication (SP) 800-37 Baseline

►Select an initial set of baseline security controls for the information system based on the security categorization; tailoring and supplementing the security control baseline as needed based on an organizational assessment of risk and local conditions.

Federal Security Standards

►NIST Special Publication (SP) 800-37 Implement

►The security controls and describe how the controls are employed within the information system and its environment of operation.

Federal Security Standards

►NIST Special Publication (SP) 800-37 Assess

►Assess the security controls using appropriate assessment procedures to determine the extent to which the controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements for the system.

Federal Security Standards

►NIST Special Publication (SP) 800-37 Authorize

►Information system operation based on a determination of the risk to organizational operations and assets, individuals, other organizations, and the Nation resulting from the operation of the information system and the decision that this risk is acceptable.

Federal Security Standards

►NIST Special Publication (SP) 800-37 Monitor

►The security controls in the information system on an ongoing basis including assessing control effectiveness, documenting changes to the system or its environment of operation, conducting security impact analyses of the associated changes, and reporting the security state of the system to designated organizational officials.

Federal Security Standards

►NIST Special Publication (SP) 800-37►The final step in the cycle, Monitor, is of

particular importance because it evaluates the effectiveness of a security control. But what if you only performed this evaluation periodically—for example, to satisfy a quarterly or annual audit for a regulation or other compliance related demand? Unfortunately, it could be months or even a year before you’d realize that the security control was not functioning as intended.

Federal Security Standards

►NIST Special Publication (SP) 800-37 Assess

►Assess the security controls using appropriate assessment procedures to determine the extent to which the controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the security requirements for the system.

Some Things to Monitor

►Patch management►Network management tools►Security tools such as:

Change management Configuration management Log monitoring Vulnerability scanning solutions

►10 signs that you aren't cut out for IT 1: You lack patience 2: You have no desire to continue

your education

10 Signs you aren’t cut out for IT

Logs?

►Syslog is the predominant standard for computer system logging

►Microsoft, in its infinite wisdom chose their own called “Windows Event Log”. There are several converters to convert

the Windows Event Log to the Syslog standard.

Log Log Log

►Many incidents can be readily revealed with a bit of logging and analysis those logs.

Logs

►Solutions Almost everything that has a log should

have the log turned on. Logs should include:

►Date/time►Source IP►Destination IP►Port►Etc

Logs

►Solutions Use standard SYSLOG entries or use

software that converts logs to a common log format.

Store logs for a while – space & DVDs are cheap

Create systems & procedures for analyzing logs.►These systems should have ‘normal’ items and

‘abnormal’ items

Logs

►Solutions All remote access logging:

►should be in detail►Should be rigorously analyzed.

All security alerts should be logged.►Workstation►Servers►Devices

Logs

►Solutions Use unified time

►This allows logs to be matched up across many devices and / or networks.

Border devices►Should log verbosely►Should log all traffic

Blocked Allowed

Logs

►Solutions Logs should be secured Logs should be exported & saved on Write

Once devices.or Logs should be written to dedicated

logging servers. The dedicated logging servers with

separate security credentials

Logs

►Solutions Test the logs and review after:

►Normal / acceptable traffic►Push the system►Attempt to penetrate the network.

Inside Outside

►Compare and correlate the data on all of the logs for validity.

Logs

►Solutions Review

►Logs everyday►Use automated tools to analyze large amounts

of data. Test

►Attack a system►Test the response time.

Discovery Action taken to attack

Log Review Tools

►Windows -> Syslog conversion Snare agent (

intersectalliance.com/projects/index.html) and ProjectLasso remote collector (sourceforge.net/projects/lassolog) are used to convert Windows Event Logs into syslog, a key component of any log management infrastructure today (at least until Visa/W7 log aggregation tools become mainstream

►10 signs that you aren't cut out for IT 1: You lack patience 2: You have no desire to continue

your education 3: You refuse to work outside 9-to-5

10 Signs you aren’t cut out for IT

Database Activity Management

►Database Activity Monitoring (DAM) is a database security technology

for monitoring and analyzing database activity that operates independently of the database management system (DBMS)

It does not rely on any form of native (DBMS-resident) auditing or native logs such as trace or transaction logs.

DAM is typically performed continuously and in real-time.

Database Activity Management

►Add prevention and you get (DAMP) This extension to DAM goes beyond

monitoring and alerting to also block unauthorized activities.

DAM helps organizations address compliance:►HIPAA►PCIDSS►Sarbanes-Oxley (SOX)►NIST 800-53

Database Activity Management

►Features include: Event aggregation Correlation Reporting Auditing

►Does not require access to native database audit functions

Database Activity Management

►Privileged User Monitoring: Monitoring privileged users:

►DBAs►Sysadmins►Developers

who typically have unfettered access to corporate databases

Protects against external and internal threats

Database Activity Management

Monitors all activities and transactions Identifies anomalous activities

►Viewing sensitive data►Creating new accounts

(with superuser privileges?)

►Adding or Deleting tables

Database Activity Management

►Most organizations have perimeter protection

►The next need is to monitor and protect privileged user accounts

Database Activity Management

►There is a high correlation between and protection from the insider.

►Privileged users are capable of: Stored procedures Triggers Views

Database Activity Management

►Targeted attacks frequently result in attackers gaining privileged user credentials: Monitoring of privileged activities is also

an effective way to identify compromised systems.

Database Activity Management

►Privileged user monitoring helps ensure: Data Privacy Integrity

Implementation of Logs

►Procedures and Tools to Implement and Automate this Control Most Everything allows logging Evaluate what is and what is not being

logged.►compare them with the asset inventory

Implemenatation of Logs

►Manual inspection analyze logs on individual devices correlation (SIEM) tools can make audit

logs far more useful

Implementation of Logs

►SIEM & Consolidation tools can be quite helpful in identifying subtle attacks. These tools are not a replacement for

skilled information security personnel and system administrators.

Even with automated log analysis tools, human expertise and intuition are often required to identify and understand attacks.

Log Measurement - Manual

►Item: Network time protocol (NTP) Measurement: Confirm that NTP is

being used to synchronize time for all devices and that all clocks are in synch.►Pass or fail.

Log Measurement - Manual

►Item: Vulnerability scanner Measurement: Run a non intrusive

vulnerability scanner against random servers.►Review logs to determine whether the

information appeared in the logs.Pass or fail.

Log Measurement - Manual

►Item: Security Event Information Management system (SEIM / SIEM / etc) Measurement: Correlate logs to a

central source and determine that all servers are properly logging.►Compare to inventory list►Start @ 100% and back off 5% for each

device not logging.

SIEM

►security information and event management

►security incident and event management

SIEM

►SIEM is a computerized tool used on data

networks to centralize the storage and interpretation of logs, and events.

The logs and events are generated by other hardware and software products on the network.

SIEM

►SIEM should include: Gathering the logs

SIEM

►SIEM should collect logs from: Syslogs Firewall logs IDS logs Windows server logs Database logs Web server logs Application logs.

SIEM

►SIEM may need to: Handle multiple data centers Collect data centrally

SIEM

►SIEM Reports: Correlating

►Their big value is in the correlation of data from multiple sources in multiple formats.

Regulatory Trouble-shooting Investigating Alerting

SIEM

►SIEM can also detect: Distributed attacks Complicated attack paths Insider abuse

►As well as: Normal network performance failures

►(Requires a capable analyst)

Commercial Software Products

►CorreLog Enterprise Server – software $5k

►ManageEngine EventLog Analyzer $350 / 10 sources

►NitroSecurity NitroView $29k►Prism Microsystems EventTracker $30k►Tripwire Log Center $19k

►10 signs that you aren't cut out for IT 1: You lack patience 2: You have no desire to continue

your education 3: You refuse to work outside 9-to-5 4: You don’t like people

10 Signs you aren’t cut out for IT

NMap

►Nmap A security scanner originally written by

Gordon Lyon (aka Fyodor Vaskovich)►1) used to discover hosts►2) user to discover services on hosts►3) can determine;

which ports are open and closed the operating system names and versions of the listening services estimated uptime type of device presence of a firewall.

Nmap

NMap

►DEMO

►10 signs that you aren't cut out for IT 1: You lack patience 2: You have no desire to continue

your education 3: You refuse to work outside 9-to-5 4: You don’t like people 5: You give up quickly

10 Signs you aren’t cut out for IT

Windows Log Analysis

►Free Products LogZilla (code.google.com/p/php-syslog-

ng) Analyzes Syslogs Is PHP-based visual front-end

►For syslog servers Searches Reports etc

Windows Log Analysis

►Free Products Splunk ( Free for first 500mb )

Windows Log Analysis

►Free Products►OSSEC (ossec.net):

an open source tool Analyzes Real-Time:

►Unix systems►Windows servers►Network devices

Includes:►default alerting rules

Windows Log Analysis

►Free Products Snare agent & ProjectLasso (

intersectalliance.com/projects/index.html) and remote collector (sourceforge.net/projects/lassolog)►Open sourc tool►Analyzes real-time:

Windows Event Logs Syslog Network devices

Windows Log Analysis

►Free Products Log2timeline (log2timeline.net/) Analyzes Logs

►Used as an investigation tool it can create a timeline view out of raw log data

►Runs on Linux and Mac using Perl

Windows Log Analysis

►Free Products syslog-ng (

balabit.com/network-security/syslog-ng/) Open Source Analyzes

► Windows Logs by installing an agent on the server

►Syslog

SpiceWorks

►Spiceworks headquartered in Austin, Texas.

►It was formed in early 2006 to provide a Facebook-like community integrated with a free ad-supported IT: Systems Management Inventory Help desk software

SpiceWorks

►The product is designed for network administrators working in small- to mid-sized businesses and managing up to a few thousand network devices.

SpiceWorks

►SpiceWorks discovers:► Windows► Unix► Linux► Mac OS X► Routers► VOIP phones► Printers► etc.

SpiceWorks

►DEMO

►10 signs that you aren't cut out for IT 1: You lack patience 2: You have no desire to continue

your education 3: You refuse to work outside 9-to-5 4: You don’t like people 5: You give up quickly 6: You’re easily frustrated

10 Signs you aren’t cut out for IT

What exactly is WireShark?

►Wireshark is a sniffer Sniffer, Packet Analyzer, also known as a

network analyzer, protocol analyzer. A software product and / or hardware product

that has the ability to intercept all network traffic and allow for the analysis of the packets of data contained in that traffic.

What exactly is WireShark?

►Originally named Ethereal, Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education

►It runs on all popular computing platforms, including Unix, Linux, and Windows.

What is a sniffer?

►Protocol Analyzer Who is talking to who What they are saying Header and Overhead Payload Problems

Use of a sniffer for security:

►Basic Information: 7 layer OSI model

►Application►Presentation►Session►Transport►Network►Datalink►Physical layers

Use of a sniffer for security:

►Many believe that you look at everything from the bottom up just like:

Building a building Math Employment, etc

►But, that is not always true…. With a house you start with the overall design….

►In the case of sniffing It is best to start somewhere in the middle, usually

at TCP or ICMP, then move down or up based upon what you discover and what you are looking for.

Requirements of a Sniffer

►Hubs►Switches►Switches

Port Mirroring Switched Port Analyzer (SPAN) Roving Analysis Port (RAP)

►Local Connection must operate in promiscuous mode

Example Sequence 1

►DHCP request and response►ARP for gateway►DNS “A” request and response►Web session setup►Payload delivery

DHCP

DHCP: src & dst

DHCP: IP Component

DHCP: UDP Component

DHCP: BootStrap Parameters

DHCP: First Packet Results

DHCP: Second Packet Results

DHCP: Third Packet Results

DHCP: Fourth Packet Results

►10 signs that you aren't cut out for IT 1: You lack patience 2: You have no desire to continue your

education 3: You refuse to work outside 9-to-5 4: You don’t like people 5: You give up quickly 6: You’re easily frustrated 7: You can’t multitask

10 Signs you aren’t cut out for IT

DNS

►DNS is the internet’s system to translate the human readable name such as EvidenceSolutions.com to the actual IP address of the desired site.

DNS: Request

DNS: Response

We get to the website: syn

We get to the website: syn & ack

Website final: ack

Website Sends Graphics

How to use the product….

►Basics out of the way Start at the bottom & work your way up

►Works great for troubleshooting

Or start at the top an work your way down►Not so good for security- top down is best here

►Real world virus example Multiple protocols Disguised traffic Peel the onion to analyze

Real World

STATISTICS- PROTOCOL HIERARCHY TCP CONVERSATIONS

► Compare to your list of valid IP’s► Compare to your list of valid protocols► Compare with what you have seen in

the past►Drill down into any anomalies or

unusual instances

Real World

FTP Traffic High…

FTP Traffic High… to .10

FTP Traffic High… between .10 & .25

.25 is the culprit but what is it doing

Conclusion for this scan:

►Normally, a port scan shows up very loudly and easily

► In this case, .2 was controlling .10 via a remote trojan on port 25 (mail) . .10 was passing the instructions on to .25 via port 21 (ftp)

►All of this trojan traffic passes firewall rules and doen’t get a second glance.

►You would have busted .25 for the port scan, but left .10, .25 infected and the master of it all, .2, is still at large.

Conclusion for this scan:

References

► Cool sample traffic, including security issues: http://wiki.wireshark.org/SampleCaptures

#head-6c6fb4051dfbe9b992057ea1533eb8dc85c9a13a

► Filters http://wiki.wireshark.org/DisplayFilters

► 10 signs that you aren't cut out for IT 1: You lack patience 2: You have no desire to continue your education 3: You refuse to work outside 9-to-5 4: You don’t like people 5: You give up quickly 6: You’re easily frustrated 7: You can’t multitask 8: You have dreams of climbing the corporate ladder 9: You hate technology 10: You turn off your phone at night

► By Jack Wallen; February 24, 2012

10 Signs you aren’t cut out for IT

Evalution

►I value your comments. Please fill in your evaluation form found at the end of your packet.

Scott Greene: Other topics available

► Computer Forensics► Computer Forensics for Defense Attorneys► Personal Privacy in the Information Age► High Technology: Just where is technology going?► Bypassing Security: How They Steal Company Data ► Fundamentals of Digital Forensics► Technology Forensics: Theory & Potential... is it Science or Art?► Technology Forensics: Case Examples► Technology Forensics: Intellectual property and identity theft► Technology Forensics: Hardware and Software tools / Show and Tell► Portable Devices Issues and Answers: A discussion about cell phones and the

stories they can tell.► Anti-Digital Forensics. Or is it Digital Anti-Forensics?► Data Security and Confidentiality Issues► E-mail: The digital Smoking Gun

Contact InformationScott Greene, SCFE

Evidence Solutions, Inc

866-795-7166Scott@EvidenceSolutions.com