5 Reasons Corelight Gives You Better Data for Incident Response …€¦ · The network provides...

CORELIGHT, INC. | [email protected] 1

The Need for Better Data

The network provides invaluable “ground truth” for incident response and threat hunting since virtually all attacks must cross the network. Corelight’s network traffic analy-sis capabilities come from Bro, the powerful, open-source network monitoring framework. The data output of Corelight products--Bro logs--are detailed summaries of network events organized by protocol and designed for use in incident response, threat hunting, and network forensics.

When security professionals first hear about Corelight and Bro, they often ask if their existing network components and tools already give them equivalent data. This is a natural question, but these data sources normally have significant limitations:

• Poor accessibility: data comes from many devices and tools distributed throughout the network. Some data may be aggregated in a SIEM, but the bulk often remains inaccessible to security teams.

• Poor searchability: making sense of network data, especially raw network packets (i.e., PCAP), is tedious and time-consuming. It’s difficult to uniformly search across different data sources due to varying data formats and timestamps. As a result, analysts have trouble recon-structing the chain of events that lead to compromise.

• Poor context: Low-level network logs, such as NetFlow, do not provide enough detail about a given event to answer critical questions for incident response or foren-sics. DNS logs from servers, for example, will list the query names, but lack the answers to the queries.

Organizations tend to accept these limitations as normal and unavoidable and their ability to respond to security threats suffers as a result.

INSIDE: The 5 reasons, named & explained.

5 Reasons Corelight Gives You Better Data for Incident Response and Threat Hunting

2


CORELIGHT, INC. | [email protected]

and acquiring and storing it at scale is cost-prohibitive for most organizations. Sometimes “all the data” is actually “too much data”, which is precisely why PCAP does poorly on the “Search”, “Acquisition”, and “Retention” criteria. It’s just not usable for routine needs.

Bro logs would be a straight-A student on Sanders’s report card were in not for “Acquisition”, which he assigns a “C”, noting they’re “difficult to customize” and require a lot of horsepower to collect and process. Of course, his assess-ment is based on an open-source implementation of Bro, and that’s precisely where Corelight comes in: to make the generation and delivery of Bro logs dead simple for organizations with our plug-and-play sensor. Our products are enterprise-class appliances built on the foundation of open-source Bro.

Overall, this table and Sanders’s grades nicely illustrate the “Goldilocks” quality of Bro logs: they hit the perfect sweet spot between too much network data (PCAP) and too little (NetFlow). Bro logs offer just the right amount of data, without sacrificing context or usability. Now, let’s dig into what specifically makes these logs so special.

Table 1: The Graded Value2 of Different Network Data Sources for Security Investigations

Cont

ext

Pivo

t Fie

lds

Sear

ch

Acq

uisi

tion

Rete

ntio

nBro Logs A A A C A

NetFlow D D A+ A A+

PCAP A+ A+ C D D

Firewall logs D+ D B B A+

DNS Transaction Logs F C B C B

SSL Transaction Logs F D C D B

Defining Better Data

Corelight, as a solution provider dedicated to traffic analy-sis1 built on Bro, can bridge this data gap. So what makes Corelight’s solution so much better?

1. Comprehensive - it covers dozens of network protocols in granular, actionable detail.

2. Searchable - it’s formatted, organized, and interlinked for easy, fast search.

3. Extracted Files - the platform can scalably extract all files that cross the network.

4. Dynamic Protocol Detection - the platform isn’t fooled by protocol deception or errors.

5. Script Readiness - the data is designed for machine analysis and automation.

This white paper explores these five concepts in greater detail, including specific technical examples that illus-trate the difference that Corelight’s network logs can make when compared to data that incident responders or threat hunters might otherwise use. With better data from Corelight, our customers have dramatically reduced average incident response time and boosted threat hunting and network forensic capabilities.

Since Corelight was founded by the creators and maintainers of the open-source Bro framework, we’re understandably enthused about Bro’s value as a data source, but you don’t have to take our word for it, as many advocates exist outside of our company. For example, Chris Sanders, a well-known security researcher, SOC trainer, and author of the interna-tional best seller Practical Packet Analysis, graded various types of network data for incident response against a number of different investigative grading criteria, which you can see in the table at right (Table 1).

Reconstructing what happened is the primary task of incident response, so “Context” is arguably one of the most important criteria in this table. PCAP obviously earns the highest grade on “Context” because it’s a complete copy of the traffic, but searching through that ocean of data is inefficient,

3



1. Corelight’s Data is More Comprehensive

No other data source can provide as comprehensive and actionable a picture of what’s happening on your network as Bro logs. The Bro framework currently analyzes 35+ different network protocols, and support for analysis of new protocols can be added with as much detail as the program-mer wants to provide. Bro was designed with extensibility3 in mind.

Alternatively, while you can use a patchwork of network logs from various appliances and security tools such as DNS and firewall logs, these network data sources provide evidence that’s limited in scope, difficult to cross-analyze, and often questionable in accuracy, especially when it concerns security alerts. The following example illustrates the comprehensiveness of Bro’s DNS logs compared to the logs generated by a DNS server:

Figure 1.1: Example DNS Server Log4

client 192.168.117.234#53311: view authoritative: query: example.org IN NS -EDC 192.168.36.217)

This DNS server log tells us that a specific client (192.168.117.234) on a specific port (53311) wants the host names for the name servers (NS) of a given domain (example.org). This log cannot, however, answer a funda-mental question: what was the client’s answer to this request? It can’t answer this question because the answer isn’t logged, yet DNS replies can provide vital information for the work of incident response and threat hunting.

While an incident responder could go to the PCAP files for this information, that’s not fast or easy. A Bro DNS log for an equivalent network event offers the context that’s missing from the DNS server log in a fast, accessible format. The comparative richness of the Bro DNS log is immediately apparent when you look at the data in Figure 1.2 at the top of the next column.

Figure 1.2: Example Bro Log Detail for DNS Request

1308930716.700706 CNFhPo1bq5dJD3wzJ6 172.16.238.131 54304 172.16.238.2 53 udp 27628 0.004850 maps.google.com 1 C_INTERNET 1 A 0 NOERROR F F T T 0 maps.l.google.com,74.125.225.81,74.125.225.82,74.125.225.83,74.125.225.84,74.125.225.80 5.000000,5.000000,5.000000,5.000000,5.000000,5.000000 F

This Bro log is packed with actionable details about the connection, including:

1. A time stamp: (308930716.700706) - this timestamp is precise to the microsecond, derived from the network itself via NIC hardware, which is far more accurate than a timestamp that relies on the accuracy of a DNS server that could be faulty or overloaded. Incident response is all about reconstructing a narrative, and sequence is the most critical element of a narrative, so accurate event timestamps can prove quite important.

2. A Connection Unique identifier (CUID): (CNFhPo1bq5d-JD3wzJ6) - this UID is a unique identification specific to the connection and allows an investigator to easily ‘pivot’ on this connection, expose all other logs associ-ated with the connection, and understand what came before and after. This is much more efficient than painstakingly and manually matching timestamps of disparate logs to reconstruct a single connection.

3. A more complete call detail record: (172.16.238.131 54304 172.16.238.2 53 udp). This string makes up the 5-tuple, the values that constitute the connection: the source IP & port (172.16.238.131 54304) , the desti-nation IP and port (172.16.238.2 53) and the protocol used (udp). DNS server logs omit these important specifics.

4. The round trip time: (0.004850) - measured in seconds, this value could potentially reveal a man-in-the-middle attack, network misconfigurations, and/or significant network performance problems.

4



Figure 2.1: Downloading an Image

Figure 2.2: An Invidual Packet View in Wireshark of the Image Download Event

Figure 2.3: An Aggregated Connection View in Wireshark of the Image Download Event

5. The content of the response and how long to trust the

server: (74.125.225.81,74.125.225.82,74.125.225.83,74.125.225.84,74.125.225.80 5.000000,5.000000,5.000000,5.000000,5.000000,5.000000 F) - in some attacks DNS can be manip-ulated so the victim will expect a known IP address, but will get a malicious IP address back in response, but one that the attacker causes to disappear from the client’s cache after only a few seconds, masking the deception. This specificity of this log data, however, reveals exactly what the client saw as the response, including the cache lifetime.

2. Corelight’s Data is Faster to Search

Faster search means that incidents can be resolved more quickly. This increases security team productivity and limits business risk by allowing incident responders to diagnose and remediate issues before they have a chance to spread and cause greater damage.

What makes Corelight’s Bro logs better for search? First, they’re available in a single, accessible log format, which can be exported to any SIEM or data pipeline. Second, they’re specifically format-ted and linked to enable fast search. This removes the hassle of having to ingest and cross-analyze network data from different sources where data formats and timestamps may differ.

Bro is connection-oriented, not packet-oriented, and when you search Corelight’s logs you’re generally exploring connection and application-layer data. In the realm of security it’s often the higher-layer content that matters most, and is hardest to extract from PCAP and impossible to extract from NetFlow. Consid-er that 100s or 1,000s of packets could be transmitted in the download of a single, potentially-malicious PDF. Bro assimilates and analyzes all those packets, compactly logging key elements of the file transfer (including a unique connection and file ID, as well as a hash of the file - even optionally extracting and storing the PDF itself).

To see the differentiated search capabilities of Corelight’s data here, let’s look at an example5 comparing Corelight’s Bro logs versus the equivalent PCAP data generated when downloading a single image.

Downloading the image (Figure 2.1) generates only one PCAP file, but that file, as we can see in Figure 2.2 & 2.3 where it’s opened in Wireshark for analysis, contains 328 individual packets:

5



Files.log

#separator \x09#set_separator ,#empty_field (empty)#unset_field -#path files#open 2018-01-03-12-53-37#fields ts fuid tx_hosts rx_hosts conn_uids source depth analyzers mime_type filename duration local_orig is_orig seen_bytes total_bytes missing_bytes overflow_bytes timedout parent_fuid md5 sha1 sha256 extracted#types time string set[addr] set[addr] set[string] string count set[string] string string interval bool bool count count count count bool string string string string string1515012686.583020 F14RIy3scnikLTItFa 153.65.20.98 192.168.1.128 C3ykFX1N8YyAXtlIgk HTTP 0 (empty) image/jpeg - 0.333077 - F 225860 225860 0 0 F - - - - #close 2018-01-03-12-53-37

Http.log

#separator \x09#set_separator ,#empty_field (empty)#unset_field -#path http#open 2018-01-03-12-53-37#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer version user_agent request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types#types time string addr port addr port count string string string string string string count count count string count string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] vector[string] vector[string]1515012686.517278 C3ykFX1N8YyAXtlIgk 192.168.1.128 65485 153.65.20.98 80 1 GET blogs.teradata.com /data-points/wp-content/uploads/2014/02/Open-Your-Mind-to-All-The-Data3-983x1024.jpg - 1.1 curl/7.54.0 0 225860 200 OK - - (empty) - - - - - - F14RIy3scnikLTItFa - image/jpeg #close 2018-01-03-12-53-37

When it comes to incident response, raw packets are not easily navigated or searched, and they’re largely indeci-pherable to a human analyst. While you could pick out the file name in this case (Open-Your-Mind-to-All-The-Data3-983x1024.jpg) and re-query it across a larger PCAP dataset to see where else downloads occured, that’s akin to using name tags for identification in a criminal investigation: they’re poor authenticators of identity since they can be easily manipulat-ed. Wouldn’t you rather have a unique file hash instead?

Also, in this packet view it’s not easy to check this image against third party intelligence for known maliciousness. Unlike a Bro log, an incident responder would need to manually generate a hash for the image since the PCAP file does not include it. Moreover, this PCAP file doesn’t show what the person did before and after they downloaded this image--critical questions for an incident responder to answer.

Now, let’s look at a selection of Corelight Bro logs created by the equivalent image download event. The logs on this page are three distinct Bro log types related to this download event: the connection log, the file log, and the HTTP log. The unique Bro Connection ID is higlighted in yellow and the unique Bro File ID is highlighted in green in the logs.

Conn.log

#separator \x09#set_separator ,#empty_field (empty)#unset_field -#path conn#open 2018-01-03-12-53-37#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents#types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string]1515012686.450744 C3ykFX1N8YyAXtlIgk 192.168.1.128 65485 153.65.20.98 80 tcp http 0.474749 165 226111 SF - - 0 ShADadtfF 161 8933 167 234803 (empty)#close 2018-01-03-12-53-37

6



file extraction at scale by addressing the aforementioned challenges head-on:

• Unlike solutions that rely on the protocol headers to correctly self-identify, the Bro framework attempts to fully parse the protocol to robustly identify it (see the next section for discussion of this technique).

• The optimized export pipeline Corelight has developed will not re-extract duplicate files traversing a network, allowing the sensor to scale in high throughput environ-ments without significant drops or negative impacts to other sensor analysis capabilities.

4. Corelight Can’t Be Fooled By Protocol

Deception

Most network protocols use well-known ports, but attack-ers can choose to send traffic over any port and often do this to attempt to evade discovery by hiding on non-stan-dard ports (e.g., sending their HTTP traffic over the SSH port).

Many network components and tools either simply rely on well-known ports, or use simple signatures to identi-fy protocols, looking at the protocol header and trusting it to correctly self-identify. These shortcuts have signif-icant limitations. Bro, however, uses parsing validation to analyze the content of the connection and verify the protocol in use. This can uncover situations where traffic looks like a duck, but barks like a dog, such as the following example where an attacker hid their their C&C communications inside of what, at first glance, could appear be an SSL connection. When we look at the Bro DPD log for this “SSL” traffic, however, a different picture emerges:

Figure 4.1: A Bro DPD Log for an Alleged SSL Connection

{“_path”:”dpd”,”_write_ts”:”2018-01-15T17:11:57.552839Z”,”ts”:”2018-01-15T17:11:57.552839Z”,”uid”:”CpPNAD4SAqvPZf0h-5b”,”id.orig_h”:”1.2.3.4”,”id.orig_p”:3908,”id.resp_h”:”5.6.7.8”,”id.resp_p”:443,”proto”:”tcp”,”analyzer”:“SSL”,”failure_reason”:”Inval-id version late in TLS connection. Packet reported version: 4753”}

All three logs record the image-download event. Notice the search advantages these logs offer:

1. All signal, no noise: the relevant, human-readable data summarizing the connection, file event, and HTTP stream is recorded, not 328 packets’ worth of raw PCAP.

2. The Connection UIDs and File UIDs: these unique IDs, which appear in each relevant log, are unique to Bro. The Bro file ID labels the file so that a pivot can quickly identify the other connections associat-ed with that file download. The Connection ID is also unique and enables search across all of the Bro logs related to that connection (DNS, HTTP, file, etc.).

3. File hashes in Files Log: Bro automatically generates the MD5, SHA-1, (and optionally SHA-256) hashes for all files, which allows incident responders to check the hash against any malware repositories, black-lists, or other data sources and quickly search the Bro logs for any other instances of that file crossing the network.

3. Corelight Enables File Extraction At Scale

Not all file extraction capabilities are created equal, and high-performance file reassembly is a tall order, due to the difficulties associated with:

1. correctly identifying the protocol

2. correctly identifying and reassembling the packet stream

3. correctly extracting the file(s) embedded in the applica-tion dialog

4. scaling the file extraction under heavy load

Turning on file extraction in other vendors’ products will often negatively impact the performance of other network traffic analysis features in the product. You could directly extract files via PCAP, but PCAP doesn’t auto-hash files like Bro, and storing large PCAP files can get cost-prohibitive quickly. Corelight, however, excels in

7



ers must rely on the vendor to update coverage for the latest threats. The Bro framework puts organizations in the driver ’s seat by enabling custom analysis using Bro scripts. Additionally, the 20+ year history of the Bro framework and the open-source community that has grown around it means that Corelight customers can leverage existing community scripts in their own organizations. The JA3 Bro Script6 created by John B. Althouse, Jeff Atkinson, and Josh Atkins, for example, is available for download on GitHub and creates SSL client fingerprints that organizations can use to identi-fy malicious communications over SSL:

Figure 5.1: JA3 Bro Script on GitHub

In this Bro log we can see the host (1.2.3.4) and server (5.6.7.8) SSL handshake and each hello parses correct-ly as SSL. One would expect the next packet to also parse as an SSL data packet,yet it fails, described in this Bro log as “Invalid version late in TLS connection”, which indicates that the engine started to parse the connec-tion in SSL, but then failed later on in the connection.

This Bro log has uncovered that this connection is not, in fact, an SSL connection.

It ’s actually an attacker attempting to hide a backdoor by disguising their communications as SSL so that other components in the system, like an IDS, would see it and ignore it, assuming that the subsequent traffic after the SSL handshake was unreadable. In reality, it ’s two attacker-controlled machines talking to each other using their own (non-SSL) protocol. Corelight can automatically detect this kind of protocol evasion at scale across millions of connections.

5. Corelight’s Data is Designed for

Automated Analysis

Not only can you write powerful scripts that process the output of Bro logs, but you can write scripts that run before the data is generated. This means any given Bro log can be customized to include the exact details desired, and/or entirely new Bro logs can be generat-ed. In addition, Bro also allows individuals to create custom scripts to automate analysis tasks such as threat detection and network performance monitoring.

The Bro Intelligence Framework is one of the most widely used script frameworks in the Bro community. If you specify a DNS name to monitor, for example, the script will look across all protocols and alert whenever that DNS name appears.

This scripting ability makes Bro a more flexible and extensible option for network traffic analysis because customers can decide exactly what data they want and how to analyze it. This stands in stark contrast to most threat detection and security analytics products today, where the analysis logic remains opaque and custom-

8



SUMMARY

In short, not all data derived from network traffic is created equal. Many organizations today are stuck between a rock and a hard place, between the overwhelming and unwieldy firehose of PCAP and stitching piecemeal network logs together from a variety of sources, none of which provide a holistic, usable view of activity on the network.

Between the extremes of having too much data and too little, a better alternative exists: Corelight and Bro. Compared to common network data sources, Corelight’s Bro logs hold clear and compelling advantages:

1. They provide the most comprehensive, actionable picture of a network

2. They’re fast and easy to search

3. The platform can extract files at scale without impacting other network analysis functions

4. The platform can’t be fooled by deceptive traffic

5. The data is designed for both humans and automated analysis

Corelight was founded by the creators and maintainers of the open-source Bro framework with the singular vision of offering world-class network traffic analysis. Our sensors turn network traffic into high-fidelity data streams in the form of Bro logs for your SIEM or data pipeline, helping organizations achieve faster incident response times, along with more effective threat hunting and network forensics capabilities.

CITATIONS:

1. Saldich, Alan. Corelight, Inc. Bro is an IDS. Not, It’s Not. 2018. http://www3.corelight.com/bro-is-an-ids-no-its-not2. Grading criteria and grades developed by Chris Sanders, Founder at Applied Network Defense. www.investigationtheory.com 3. Kreibich, Christian. Corelight, Inc. Extensibility as a Guiding Principle. 2017. https://corelight.blog/2017/12/06/extensibility-as-a-guiding-principle/ 4. Mens, Jan-Piet. BIND querylog: know your flags. 2011.https://jpmens.net/2011/02/22/bind-querylog-know-your-flags/ 5. Image licensed for Corelight commercial use from: www.istock-photo.com. The image download event cited in the paper came from: http://blogs.teradata.com/data-points/wp-content/uploads/2014/02/Open-Your-Mind-to-All-The-Data3-983x1024.jpg

6. Script source: https://github.com/salesforce/ja3

Data designed by incident

responders, for incident responders.

smb_mapping.log | SMB mappings

FIELD TYPE DESCRIPTIONts time Time when the tree was mapped

uid string Unique ID of the connection the tree was mapped over

id conn_id ID of the connection the tree was mapped over

path string Name of the tree path

service string The type of resource of the tree (disk share, printer share, named pipe, etc)

native_file_system string File system of the tree

share_type stringIf this is SMB2, a share type will be included. For SMB1, the type of share will be deduced and included as well.dhcp.log | DHCP lease activity

FIELD TYPE DESCRIPTIONts time Timestamp of the DHCP lease request

uid & id Underlying connection info > See conn.log

mac string Client’s hardware address

assigned_ip addr Client’s actual assigned IP address

lease_time interval IP address lease time

trans_id count Identifier assigned by client responses match

ftp.log | FTP request/reply details

FIELD TYPE DESCRIPTIONts time Timestamp of the FTP command

uid & id Underlying connection info > See conn.log

user string Username for the FTP session

password string Password for the FTP session

command string Command issued by the client

arg string Any command arguments

mime_type string File type if there’s a file transfer

file_si e count Size of transferred file

reply_code count Reply code from server in response to the command

reply_msg string Reply message from server in response to the command

data_channel record Information about the data channel (orig, resp, is passive)

fuid string File unique ID

conn.log | IP, TCP, UDP, ICMP connection details

FIELD TYPE DESCRIPTION ts time Timestamp of the first packet

uid string Unique ID of the connection

id.orig_h addr Originating endpoint’s IP address (Orig)

id.orig_p port Originating endpoint’s TCP/UDP port (or ICMP code)

id.resp_h addr esponding endpoint’s IP address ( esp)

id.resp_p port esponding endpoint’s TCP/UDP port (or ICMP code)

proto proto Transport layer protocol of connection

service string Detected application protocol, if any

duration interval Connection length

orig_bytes count Orig payload bytes from sequence numbers if TCP

resp_bytes count esp payload bytes from sequence numbers if TCP

conn_state string Connection state (see conn.log > conn_state)

local_orig bool Is Orig in Site::local nets?

local_resp bool Is esp in Site::local nets?

missed_bytes count Number of bytes missing due to content gaps

history string Connection state history (see conn.log > history)

orig_pkts count Number of Orig packets

orig_ip_bytes count Number of Orig IP bytes (via IP total length header field)

resp_pkts count Number of esp packets

resp_ip_bytes count Number of esp IP bytes (via IP total length header field)

tunnel_parents set If tunneled, connection UID of encapsulating parent(s)

orig_I2_addr string Link-layer address of the originator

resp_I2_addr string Link-layer address of the responder

vlan int The outer LAN for this connection

inner_vlan int The inner LAN for this connection

| WP005-5RWDATA-V1.0-US

5 Reasons Corelight Gives You Better Data for Incident Response …€¦ · The network provides...

Documents

Transcript of 5 Reasons Corelight Gives You Better Data for Incident Response …€¦ · The network provides...