Open Source vs. Network Attackers: What’s in your arsenal?Gary Smith, Pacific Northwest National Laboratory

A Little Context

The Five Golden Principles of SecurityKnow your systemPrinciple of Least PrivilegeDefense in DepthProtection is key but detection is a must.Know your enemy.

2

Fudd’s First Law of Opposition

All brute force attacks are based on Fudd’s First Law of Opposition.Fudd’s First Law of Opposition says, “Push something hard enough and it will fall over.”SSH Brute force attacks, in particular, have been going on for at least10 years.The attacks from 10 years ago are not fundamentally different from the ones we see now except for one difference.10 years ago it would take weeks after putting a server live on the Internet for it to start being scanned.Today, if we put a new server live on the Internet, within minutes, it starts to be scanned.

3

Do you see lines like this in your syslog?

Mar 4 03:26:56 a9 sshd[13185]: Invalid user leonob from 211.157.179.71Mar 4 03:27:01 a9 sshd[13189]: Invalid user ftpuser from 211.157.179.71Mar 4 03:29:16 a9 sshd[13320]: Invalid user oracle from 211.157.179.71Mar 4 03:29:20 a9 sshd[13324]: Invalid user bwadmin from 211.157.179.71Mar 4 03:29:30 a9 sshd[13332]: Invalid user cacti from 211.157.179.71Mar 4 03:29:35 a9 sshd[13336]: Invalid user test1 from 211.157.179.71

4

Or like this?

Mar 30 09:30:40 a9 sshd[4843]: Failed password for root from 222.45.87.22Mar 30 12:30:03 a9 sshd[6626]: Failed password for root from 50.57.69.176Mar 30 13:31:58 a9 sshd[7243]: Failed password for root from 62.76.102.36Mar 30 15:38:31 a9 sshd[8491]: Failed password for root from 190.254.21.123Mar 30 18:11:52 a9 sshd[9913]: Failed password for root from 42.62.17.250Mar 30 19:27:42 a9 sshd[10812]: Failed password for root from 61.174.51.218Mar 30 22:08:04 a9 sshd[12482]: Failed password for root from 61.147.103.21Mar 31 00:01:35 a9 sshd[13706]: Failed password for root from 192.198.80.194Mar 31 02:16:34 a9 sshd[15410]: Failed password for root from 62.76.102.38

5

Or maybe this?

Mar 15 19:22:36 a9 sshd[15420]: Invalid user admin from 61.174.51.205Mar 15 19:22:57 a9 sshd[15436]: Invalid user admin from 61.174.51.205Mar 15 19:23:16 a9 sshd[15456]: Invalid user admin from 61.174.51.205Mar 16 02:22:24 a9 sshd[19740]: Invalid user admin from 116.10.191.208Mar 16 02:22:47 a9 sshd[19756]: Invalid user admin from 116.10.191.208Mar 16 02:23:05 a9 sshd[19776]: Invalid user admin from 116.10.191.208Mar 16 02:23:33 a9 sshd[19792]: Invalid user admin from 116.10.191.208Mar 16 02:23:50 a9 sshd[19808]: Invalid user admin from 116.10.191.208Mar 16 02:24:18 a9 sshd[19837]: Invalid user admin from 116.10.191.208

6

So, where do I start?

Begin a process of moving from the center outward, creating rings of security.For instance, if the server already has a public IP, you’ll want to lock down root access immediately. In fact, you’ll want to lock down SSH access entirely, and make sure that only you can get in. Add a new user, and add it to an admin group.(preconfigured in /etc/sudoers to have access to sudo).

7

SSHD Configuration

Configure the SSH daemon to more secure:PermitRootLogin noPermitEmptyPasswords noAllowUsers user1 user2 user3…AllowGroups group1 group2 group3…Protocol 2PrintLastLog yesLoginGraceTime 1m

Reload SSH to apply the changes, and then try logging in in a new session to ensure everything worked.If you can’t log in, you’ll still have your original session to fix things up.

8

Update the System

Now that you’re the only one with access to the server, you can stop worrying about a hacker sneaking in, and breathe normally again (maybe). Chances are good that there are some updates for your server, so go ahead and run those now.Depending on the distribution, the utilities and options invoked will vary to perform an update.

9

Install a Firewall

Set up a firewall, and only allow what you need right at this moment. You can always punch another hole thru as you need it.Here’s a sample set of Iptables rules that allow a minimal set of services.

10

Sample Iptables Rules

*filter

# Set a default policy of DROP across all the tables:INPUT DROP [0:0]:FORWARD DROP [0:0]:OUTPUT DROP [0:0]

# Accept any related or established connections-I INPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT-I OUTPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow all traffic on the loopback interface-A INPUT -i lo -j ACCEPT-A OUTPUT -o lo -j ACCEPT

# Allow outbound DHCP request - Maybe you need; maybe you don’t#-A OUTPUT –o eth0 -p udp --dport 67:68 --sport 67:68 -j ACCEPT

# Outbound DNS lookups-A OUTPUT -o eth0 -p udp -m udp --dport 53 -j ACCEPT

11

Sample Iptables Rules (cont.)

# Outbound PING requests-A OUTPUT –o eth0 -p icmp -j ACCEPT

# Outbound Network Time Protocol (NTP) request-A OUTPUT –o eth0 -p udp --dport 123 --sport 123 -j ACCEPT

# Inbound SSH-A INPUT -i eth0 -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT

# Outbound email-A OUTPUT -o eth0 -p tcp -m tcp –dport 25 -m state --state NEW -j ACCEPT

# Outbound HTTP and HTTPS-A OUTPUT -o eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT-A OUTPUT -o eth0 -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT

COMMIT

12

Think you’re safe? Maybe not.

You’ve hardened your SSH daemon configuration.You’ve updated your server’s software.You’ve put in a restrictive firewall.What could go wrong now?There’s still a lot of bad actors out there who will be brute forcing or DoS’ing your SSH connection.

13

Tools/Techniques to Defend Against Brute Force SSH Attacks

Roll Your OwnFail2BanDenyhostspam_abl

14

Fail2Ban – Intrusion Prevention

Fail2ban is an open source intrusion prevention framework developed in the Python programming language. Fail2ban operates by monitoring log files such as /var/log/httpd/access_log, /var/log/auth.log, /var/log/secure etc. and bans the IP address after too many password failure attempts. It updates IPTables firewall rules to reject the IP address for a specified amount of time.

15

Configuring Fail2Ban Global Defaults

After installing fail2ban, copy /etc/fail2ban/jail.conf to /etc/fail2ban/jail.local and make your changes there.Fail2Ban Global Defaults

[DEFAULT] # "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not # ban a host which matches an address in this list. Several addresses can be # defined using space separator. ignoreip = 127.0.0.1 # "bantime" is the number of seconds that a host is banned. bantime = 600 # A host is banned if it has generated "maxretry" during the last "findtime" # seconds. findtime = 600 # "maxretry" is the number of failures before a host get banned. maxretry = 3

16

Configuring Fail2Ban SSH Monitoring

Look for the [ssh-iptables] section, configure to your site.

[ssh-iptables] enabled = true filter = sshd action = iptables[name=SSH, port=ssh, protocol=tcp] sendmail-whois[name=SSH, dest=root, sendername=Fail2Ban] logpath = /var/log/secure maxretry = 5

Then, start up fail2ban.

17

How do you know it’s working?

At startup, Fail2Ban sends a “starting” email message like this to the designated recipient(s) like this:

Hi,The jail SSH has been started successfully.Regards,Fail2Ban

When Fail2Ban takes action, it sends an email to the designated recipient(s) like this:

Hi,The IP 117.27.158.78 has just been banned by Fail2Ban after 5 attempts against SSH.Here are more information about 117.27.158.78:Regards,Fail2Ban

18

How can I tell if it’s working?

You can also do iptables –nL

Chain fail2ban-SSH (1 references)target prot opt source destinationREJECT all -- 117.27.158.78 0.0.0.0/0 reject-with icmp-port-unreachableREJECT all -- 115.238.73.16 0.0.0.0/0 reject-with icmp-port-unreachableREJECT all -- 117.27.158.95 0.0.0.0/0 reject-with icmp-port-unreachable

19

Some Iptables Magic

You can restrict the number of connections used by a single IP address to your server using iptables.Only allow 4 ssh connections per client system:

iptables -A INPUT -p tcp --syn --dport 22 -m connlimit --connlimit-above 4 –j DROP

You can limit the number of connections per minute. The following example will drop incoming connections if an IP address makes more than 10 connection attempts to port 22 within 60 seconds.

iptables -A INPUT -p tcp –dport 22 -i eth0 -m state --state NEW -m recent --set iptables -A INPUT -p tcp –dport 22 -i eth0 -m state --state NEW -m recent --update –-seconds 60 -–hitcount 10 –j DROP

20

How do I know it’s working?

Use the following shell script to connect to your SSH server at 192.168.100.101:

#!/bin/bash IP=“192.168.100.101”PORT=”22" for i in {1..100} do

# do nothing just connect and exit echo "exit" | nc ${IP} ${PORT};

done

21

Other Scary Stuff From the Internet You May Be Missing

Microsoft SQL Server communication attemptsMS Terminal Server communication attemptsVNC communication attemptsPCAnywhere communication attemptsSCAN UPnP communication attemptsMicrosoft PPTP communication attemptsHP Web JetAdmin communication attemptsP2P napster communication attemptsRadmin Default install options attemptsReal Audio Server communication attemptsP2P Napster Client Data communication attempts

To protect your against these attempts, you need an intrusion detection/protection system.

22

Intrusion Detection and Log Analysis with psad and fwsnort

psad (Port Scan Activity Detector) is a collection of two lightweight system daemons that run on Linux machines and analyze iptables log messages to detect port scans and other suspicious traffic.fwsnort parses the rules files included in the SNORT intrusion detection system and builds an equivalent iptables ruleset for as many rules as possible.When psad combines with fwsnort and the Netfilter string match extension, psad is capable of detecting many attacks described in the Snort rule set that involve application layer data.psad and fwsnort can be configured to auto-block scanning IP addresses via IPTables/IP6Tables and/or tcpwrappers based on scan danger level.

23

psad Status Output

Top 50 signature matches:"MISC Microsoft SQL Server communication attempt" (tcp), Count: 186, Unique sources: 76, Sid: 100205"MISC MS Terminal Server communication attempt" (tcp), Count: 99, Unique sources: 67, Sid: 100077"ICMP PING" (icmp), Count: 85, Unique sources: 38, Sid: 384"MISC VNC communication attempt" (tcp), Count: 37, Unique sources: 19, Sid: 100202"SCAN UPnP communication attempt" (udp), Count: 9, Unique sources: 3, Sid: 100074

Top 25 attackers: 188.94.139.234 DL: 5, Packets: 29, Sig count: 6 206.126.18.30 DL: 5, Packets: 47, Sig count: 6 218.77.79.34 DL: 5, Packets: 26, Sig count: 6 93.174.93.51 DL: 5, Packets: 26, Sig count: 2 110.248.246.66 DL: 4, Packets: 22, Sig count: 0

Top 20 scanned ports: tcp 5984 7620 packets tcp 23 449 packets tcp 5000 370 packets tcp 25151 195 packets tcp 1433 189 packets

24

Psad Status Output cont.

iptables auto-blocked IPs:93.174.93.51 (unlimited timeout)218.77.79.34 (unlimited timeout)

Total protocol packet counters:icmp: 106 pktstcp: 23769 pktsudp: 191505 pkts

IP Status Detail:

SRC: 188.94.139.234, DL: 5, Dsts: 1, Pkts: 29, Total protocols: 2, Unique sigs: 1, Email alerts: 0

DST: 192.101.102.221, Local IP Scanned ports: TCP 21-3389, Pkts: 25, Chain: INPUT, Intf: eth1 Total scanned IP protocols: 2, Chain: INPUT, Intf: eth1 Signature match: "MISC MS Terminal Server communication attempt" TCP, Chain: INPUT, Count: 2, DP: 3389, SYN, Sid: 100077

25

Reports: The Final Frontier?

At some point, you (or your boss) will want to know more than “is it working?” You (or your boss) will want some kind of reports.The problem with reporting security-related data is two-fold.Problem #1: What do you represent?Problem #2: How do you represent it?

26

IBM Word-Cloud Generator

IBM Word Cloud Generator is a Java application that can quickly and easily produce an image file giving more preponderant prominence to words that appear more frequently in the source text.The application uses a configuration file to control all of the settings that affect the output, such as font, layout, the treatment of stop-words, etc.Sample invocation:

java -jar ibm-word-cloud.jar -c examples/configuration.txt -w 800 -h 600 < examples/macbeth.txt > example.png

Instead of using “Macbeth” as the source, let’s use all the invalid user names we collected in our log file as input to the IBM WCG.

27

Invalid User Word Cloud

28

Invalid User with IP Address Word Cloud

29

Where are they coming from?

All those IP addresses hitting your site, where are they located?What if you could convert a IP address to a geographical location?Maxmind (maxmind.com) provides IP geolocation and fraud prevention services and Open Source APIs and a database to convert an IP address to a geographical location. The database is updated once a month.The software can converts 192.96.206.223 to Manassas, Virginia, USA at 38.7462 latitude, -77.4903 longitude.Now you can create a table like this:

30

IP Addresses / Geolocation Table

108.177.134.82 33.67 -111.95 Phoenix AZ United States108.186.21.242 37.39 -122.02 Sunnyvale CA United States110.45.145.108 37.60 126.98 Seoul 11 Korea, Republic of115.79.32.115 19.83 105.25 Dung 34 Vietnam116.48.143.29 22.32 114.18 Kowloon 00 Hong Kong121.125.64.181 37.60 126.98 Seoul 11 Korea, Republic of123.30.185.245 21.03 105.85 Hanoi 44 Vietnam125.16.64.38 28.60 77.20 New Delhi 07 India134.147.203.115 51.48 7.22 Bochum 07 Germany141.212.121.195 42.29 -83.71 Ann Arbor MI United States142.0.79.228 25.83 -80.30 Miami FL United States169.199.89.71 37.95 -122.08 Pleasant Hill CA United States172.245.46.16 42.99 -78.73 Buffalo NY United States173.160.57.166 39.65 -104.99 Englewood CO United States173.208.140.149 39.11 -94.57 Kansas City MO United States173.208.140.150 39.11 -94.57 Kansas City MO United States173.220.109.11 40.43 -74.42 East Brunswick NJ United States173.220.12.34 40.67 -73.54 Bellmore NY United States173.231.43.142 34.05 -118.26 Los Angeles CA United States177.36.92.55 -22.85 -51.77 Nossa Senhora Das Graças 18 Brazil

This is nice but it lacks pizazz or panache. How about this instead?

31

The World View

32

Psad and Gnuplot

Psad interfaces with Gnuplot.Psad parses IPTables log data and builds both a data file and a directives file for Gnuplot.Various counting modes are supported across different  time scales.Graphing criteria can include IPTables field names including negation.Unfortunately, Gnuplot works best with integer data, so IP addresses need to be translated into integer equivalents.

33

Graphing a Month’s Activity to Find Port Scans

34

Graphing a Month’s Activity by Port

35

Conclusions

There are lot of bad actors out there using attacks based on Fudd’s First Law of Opposition to get into your systems.There are a lot of Open Source tools and techniques to thwart the efforts of the bad actors.After assessing your risk profile, deploy the appropriate mitigations to limit your exposure.

36

References

Fudd's First Law of Opposition: "We're All Bozos on This Bus", Firesign TheatreSSHD Config: http://www.openssh.com/cgi-bin/man.cgi?query=sshd_configIptables: http://www.netfilter.org/Fail2ban: http://www.fail2ban.org/Denyhosts: http://denyhosts.sourceforge.net/pam_abl: http://pam-abl.sourceforge.net/psad/fwsnort: www.cipherdyne.org

37

References

IBM Word Cloud: https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?source=AW-0VWGeo-IP: http://www.maxmind.com/Xgeolocate: http://www.atgc.org/geolocate_1_2/Gnuplot: http://www.gnuplot.info/

38

Questions?

39

Gary SmithInformation System Security Officer, Molecular

Science Computing, Pacific Northwest National Laboratory

Richland, WAgary.smith@pnnl.gov