DeVera 1

The need for intelligent monitoring in progressive computer systems

Aaron DeVera, Fordham University

Big Data Symposium - SYMP 0009

Final Report - May 2015

Abstract

The context of this paper is defined by the knowledge and information gathered from

the guest lecturers during the Spring 2015 Big Data and Cybersecurity symposium.

From the content of these lectures I will discuss an emerging issue discussed, the

automation of real time data monitoring. Data monitoring, and the automation of data

monitoring for the purpose of creating intelligent alerts, can be an extremely valuable

tool to combat threats in cyberspace, and I will briefly describe ideal approaches to

monitoring that would be useful to defend a network of computers. Finally I will provide

a walkthrough of my design for a simple data monitoring application, dirmon, its

capabilities in detecting intrusions automatically, and what role such an application

would play as a tool of a forward-thinking system administrator.

Background

The many guest lecturers featured for the symposium have provided a unique inside look

into the emergence of two strong industries: data analysis technology and cybersecurity

technology. These two industry areas are being merged into a newcutting edge; two

guest lecturers at the symposium provided a look into what is being done in this space.

Arif Khan shared with us the process that he and his peers at FireEye and FireEye

Mandiant use when they are called into a firm that has been hacked. The tools that

FireEye investigators use are for the most part monitoring tools, which they install at

endpoints within a company to create a topography of infected or non-infected

computers. Arif also described some of the scanning tools they use, essentially turning

every computer on the company topography into a sensor looking for malicious contact

that can be reported back to a command center.

DeVera 2

Tim Kropp from ISE had a very similar approach. Without a FireEye budget, he

described how he and his team developed a similar network of sensors to that of

FireEye, and showed how attacks on a large pool of computers are related using shared

data collected from the monitoring capabilities on the sensor-enabled computers. The

visualization and analysis of monitoring is an important element to understanding the

threats facing a firm or organization; it is not just enough to monitor systems. A

progressive system administrator who wishes to use a “smart” automated monitoring

system would want to engineer a system that favors actionable intelligence over

potentialities and would want to engineer a response that acts upon good analysis of all

the information supplied.

Intelligent monitoring for security

Monitoring changes in a computer system is a typical thing a system administrator may

want to do for several reasons beyond security, such as version control. However, the

monitoring of commands and behavior from users and external connections are often

targeted, and not actively monitored. This is the cutting edge of security monitoring; a

blend of cybersecurity tools and techniques being used in a large-scale data analysis.

There are a select number of firms and vendors in this space, offering products that

allow for big data analysis upon monitoring logs.

The velocity of monitoring data can be troublesome, with many events being

generated at a quick rate, with potentially very few events having actionable intelligence

value. Intelligent data monitoring is a developing field that can facilitate the analysis of

monitoring for security purposes. Using a machine learning approach, an automated

control paradigm of a monitoring system could potentially learn what is “normal” in the

way that many system administrators and forensic investigators do. With the ability to

build new controls upon preexisting each terms, an automated control can build layers

that will make intrusions more obvious to the monitoring system and the system

administrator.

DeVera 3

Monitoring solutions

An ideal system protected by intelligent monitoring would be one where every endpoint

has an embedded sensor, reporting back to a central system run by a system

administrator. Each endpoint sensor can be as simple as some lines of code that enables

unchangeable and encrypted reporting of the endpoint’s status back to the central

command. The central command could then map out the network topography, and

begin to implement intelligent control paradigms that can use machine learning to

understand “normal” network behavior. In the event of an intrusion, this paradigm

should be able to enact and automated response. For instance, in the event that a

phishing attack is spreading amongst a part of the network, the intelligent control

paradigm should be able to [1] recognize the attack, [2] cutoff network and physical

communication on the endpoint machines, [4] capture snapshots of the endpoint

machines states for further analysis. Depending on the atomic implementation of the

endpoint sensor, the quarantine of malicious behavior could maybe be isolated within a

single machine’s disk or filesystem.

Technical overview of a simple monitoring program

As an exercise in intelligent monitoring, I have constructed a program to demonstrate

the possibilities of automated monitoring systems. This program, dirmon, can be

activated while in a folder/directory. Once activated, it will watch that directory for

bash or shell command line behavior that concerns it. For instance, let’s assume I have a

folder named “myFolder” where dirmon is activated. If an intruder was attempting to

enter or even read contents “myFolder” using the cd or ls commands, respectively, alerts

would be generated within the dirmon monitoring system. dirmon then determines of

the behavior is verifiably affecting or reading info containing “myFolder,” and if true, a

popup window warns the user of the attack magnitude and type of attack. The dirmon

program was developed for Unix systems, and should work on both Linux and OSX. It

can be activated as a daemon program so that it runs in the background without

DeVera 4

disturbing normal user activity on the machine. Attached to this report is the dirmon

source code, which can also be found on GitHub under my account aaronsdevera.

Conclusion

Ultimately the future of intelligent monitoring will be determined by the demand for

such tools as monitoring large networks becomes more of a hassle. Intelligent monitoring

is the next logical step to system monitoring, and will only get more useful with the

advent of better large-scale data collection and analysis tools.

5/7/15, 3:15 AM

Page 1 of 3https://raw.githubusercontent.com/aaronsdevera/dirmon/master/dirmon.py

# dirmon.py# A directory tracker written by Aaron DeVera# May 6, 2015## USAGE# Just navigate to the directory you want to track then run the command:# > python dirmon.py## REQUIREMENTS# This program requires a bash_history merge across tty sessions.# > export HISTCONTROL=ignoredups:erasedups # > shopt -s histappend# > export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"## TO DAEMONIZE THIS SCRIPT# use command:# > python dirmon.py &# NOTE: you should disable all "print" commands in the daemon script in the "difference_hunter" module

import timeimport psutilimport osimport reimport easygui

cur_dir_path = os.getcwd()splitter = re.compile(r'/')cur_dir = splitter.split(cur_dir_path)cur_dir = '/' + cur_dir[len(cur_dir)-1]splitter = re.compile(r'\n')count = 0

home = os.path.expanduser('~')print '\n============================================================'print 'dirmon initiated!\nLogs kept at ~/.dirmon and ~/.dirmon_origin'print 'Initiating tracker for current directory:'+cur_dirprint 'At path:' + cur_dir_pathprint '============================================================'

def prime_logger(): bash_history_raw = open(home+'/.bash_history', 'r') bash_history_text = bash_history_raw.read() with open('./.dirmon_origin','w') as f: f.write(bash_history_text) f.close()

def bash_history(): # import bash history bash_history_raw = open(home+'/.bash_history', 'r') bash_history_text = bash_history_raw.read() bash_history_array = splitter.split(bash_history_text) return bash_history_array def dirmon_origin(): # open dirmon origin file dirmon_origin_raw = open('./.dirmon_origin','r+w') dirmon_origin_text = dirmon_origin_raw.read() dirmon_origin_array = splitter.split(dirmon_origin_text) return dirmon_origin_array

def difference_scan(x,y): # Tally counts in x seen_count = {} differences = 0 for i in x: if i not in seen_count: # First time seeing this number. seen_count[i] = 1 else: # Number has been seen. seen_count[i] += 1

for j in y: if j not in seen_count: seen_count[j] = 1 elif seen_count[j] > 1: seen_count[j] -= 1

5/7/15, 3:15 AM

Page 2 of 3https://raw.githubusercontent.com/aaronsdevera/dirmon/master/dirmon.py

else: seen_count[j] += 1 for k in seen_count: if (k in x and k not in y) or (k in y and k not in x): while seen_count[k] > 0: seen_count[k] -= 1 elif seen_count[k] == 1: seen_count[j] += 1 differences += 1 return differences def difference_array(x,y): # Tally counts in x seen_count = {} results = [] for i in x: if i not in seen_count: # First time seeing this number. seen_count[i] = 1 else: # Number has been seen. seen_count[i] += 1 for j in y: if j not in seen_count: seen_count[j] = 1 elif seen_count[j] > 1: seen_count[j] -= 1 else: seen_count[j] += 1 for k in seen_count: if (k in x and k not in y) or (k in y and k not in x): while seen_count[k] > 0: results.append(k) seen_count[k] -= 1 elif seen_count[k] == 1: results.append(k) return results

def difference_hunter(count): flag = difference_scan(dirmon_origin(),bash_history()) # '[' + str(count) + ']' + ' ' + 'Tracking changes in ' + cur_dir_path + ':' + str(flag) if flag!=0 and flag>0: dirmon_array = difference_array(dirmon_origin(),bash_history()) dirmon = '' for f in dirmon_array: dirmon = dirmon + '\n' + f with open('./.dirmon','r+w') as f: f.write(dirmon) f.close return flag def dir_intrusion(current_directory,dirmon): results = [] for each in dirmon: if current_directory not in each: print 'Unlikely threat detected with command:' + each else: print 'Possible threat Detected with command:' + each results.append(each) return results

def run(): prime_logger() count = 0 flag_log =0 while True: flag = difference_hunter(count) if flag!=flag_log and flag>flag_log: with open('./.dirmon','r') as f: df = f.read() dirmon = splitter.split(df) intrusions = dir_intrusion(cur_dir,dirmon) easygui.msgbox('Found '+str(len(intrusions))+' intrusions into '+cur_dir+'\n'+str(intrusions), title='dirmon') flag_log=flag

5/7/15, 3:15 AM

Page 3 of 3https://raw.githubusercontent.com/aaronsdevera/dirmon/master/dirmon.py

time.sleep(.5) count += 1 if __name__ == "__main__": run()